Jul 16 21:48:10.539835 ip-10-0-131-231 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 21:48:10.539972 ip-10-0-131-231 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 21:48:10.539982 ip-10-0-131-231 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 21:48:10.540269 ip-10-0-131-231 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 21:48:20.722732 ip-10-0-131-231 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 21:48:20.722753 ip-10-0-131-231 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e0c5968213a24ad79a85b4b64d0c1cf8 -- Jul 16 21:49:58.631151 ip-10-0-131-231 systemd[1]: Starting Kubernetes Kubelet... Jul 16 21:49:58.993311 ip-10-0-131-231 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:49:58.993311 ip-10-0-131-231 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 21:49:58.993311 ip-10-0-131-231 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:49:58.993311 ip-10-0-131-231 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 21:49:58.993311 ip-10-0-131-231 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:49:58.995032 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:58.994915 2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 21:49:59.001324 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001304 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:59.001324 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001323 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:59.001324 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001326 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001329 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001332 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001335 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001338 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001341 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001353 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001358 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001361 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001365 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001367 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001370 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001374 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001377 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001380 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001382 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001385 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001389 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001393 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:59.001414 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001395 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001398 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001401 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001404 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001407 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001410 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001413 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001415 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001418 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001421 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001423 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001426 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001430 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001433 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001436 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001439 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001442 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001446 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001448 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:59.001892 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001451 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001453 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001456 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001458 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001460 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001463 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001465 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001467 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001470 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001472 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001474 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001476 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001479 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001481 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001484 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001487 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001490 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001493 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001496 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001498 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:59.002375 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001501 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001503 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001507 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001511 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001516 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001521 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001528 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001532 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001537 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001540 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001542 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001546 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001548 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001551 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001553 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001556 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001558 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001561 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001563 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001566 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001568 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:59.002882 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001570 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001573 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001575 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001578 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001580 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001583 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001586 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001589 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001592 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001595 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001599 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001604 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001608 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001613 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001615 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001618 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001621 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.001623 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002424 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002436 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:59.003398 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002440 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002443 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002446 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002449 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002453 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002457 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002460 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002463 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002465 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002468 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002470 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002473 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002475 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002478 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002480 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002483 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002486 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002488 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002491 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002493 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:59.003922 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002495 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002498 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002500 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002503 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002505 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002508 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002510 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002513 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002515 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002518 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002520 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002528 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002532 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002534 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002537 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002539 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002542 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002544 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002548 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:59.004410 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002552 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002555 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002557 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002560 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002562 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002565 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002569 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002573 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002576 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002579 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002581 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002584 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002587 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002589 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002592 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002595 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002597 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002600 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002602 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:59.004878 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002605 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002607 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002610 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002612 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002616 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002619 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002624 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002626 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002629 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002631 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002634 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002636 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002639 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002641 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002644 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002647 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002649 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002652 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002654 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002657 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:59.005339 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002659 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002661 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002664 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002666 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002669 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002671 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002674 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002676 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002679 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002682 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002684 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002687 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002691 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002695 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002698 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002700 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002703 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002705 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.002708 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003108 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:59.005848 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003114 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003116 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003119 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003121 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003124 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003126 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003129 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003131 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003134 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003137 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003139 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003141 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003144 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003147 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003149 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003152 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003155 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003160 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003162 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003165 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:59.006343 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003168 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003170 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003173 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003175 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003178 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003180 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003183 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003185 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003188 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003190 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003195 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003198 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003201 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003204 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003207 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003210 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003212 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003214 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003217 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003219 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:59.006834 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003222 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003224 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003227 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003229 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003231 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003234 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003236 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003239 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003241 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003245 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003248 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003250 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003252 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003255 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003258 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003260 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003263 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003265 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003267 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:59.007318 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003270 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003273 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003276 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003278 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003281 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003284 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003286 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003289 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003291 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003294 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003296 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003299 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003301 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003304 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003306 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003310 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003312 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003315 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003317 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003320 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:59.007821 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003322 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003326 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003329 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003332 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003334 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003337 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003339 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003342 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003344 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003346 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003349 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003351 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003353 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003356 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003358 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003362 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003366 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003369 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.003373 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003446 2558 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 21:49:59.008545 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003453 2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003459 2558 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003464 2558 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003468 2558 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003471 2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003475 2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003480 2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003485 2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003488 2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003491 2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003503 2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003506 2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003509 2558 flags.go:64] FLAG: --cgroup-root="" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003512 2558 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003516 2558 flags.go:64] FLAG: --client-ca-file="" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003519 2558 flags.go:64] FLAG: --cloud-provider="external" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003522 2558 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003526 2558 flags.go:64] FLAG: --cluster-domain="" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003529 2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003532 2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003535 2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003538 2558 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003541 2558 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 21:49:59.009046 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003544 2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003547 2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003550 2558 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003553 2558 flags.go:64] FLAG: --contention-profiling="false" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003556 2558 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003559 2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003562 2558 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003565 2558 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003570 2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003573 2558 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003576 2558 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003579 2558 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003582 2558 flags.go:64] FLAG: --enable-server="true" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003585 2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003589 2558 flags.go:64] FLAG: --event-burst="100" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003592 2558 flags.go:64] FLAG: --event-qps="50" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003596 2558 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003599 2558 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003602 2558 flags.go:64] FLAG: --eviction-hard="" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003606 2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003609 2558 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003611 2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003614 2558 flags.go:64] FLAG: --eviction-soft="" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003618 2558 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003621 2558 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 21:49:59.009629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003624 2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003627 2558 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003630 2558 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003633 2558 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003635 2558 flags.go:64] FLAG: --feature-gates="" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003639 2558 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003642 2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003645 2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003648 2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003651 2558 flags.go:64] FLAG: --healthz-port="10248" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003654 2558 flags.go:64] FLAG: --help="false" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003657 2558 flags.go:64] FLAG: --hostname-override="ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003660 2558 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003662 2558 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003665 2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003668 2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003672 2558 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003675 2558 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003678 2558 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003681 2558 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003683 2558 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003686 2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003690 2558 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003693 2558 flags.go:64] FLAG: --kube-reserved="" Jul 16 21:49:59.010371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003696 2558 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003699 2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003702 2558 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003704 2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003707 2558 flags.go:64] FLAG: --lock-file="" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003710 2558 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003715 2558 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003718 2558 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003723 2558 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003726 2558 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003729 2558 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003732 2558 flags.go:64] FLAG: --logging-format="text" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003735 2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003738 2558 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003741 2558 flags.go:64] FLAG: --manifest-url="" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003744 2558 flags.go:64] FLAG: --manifest-url-header="" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003748 2558 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003751 2558 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003754 2558 flags.go:64] FLAG: --max-pods="110" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003758 2558 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003760 2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003763 2558 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003766 2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003769 2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003772 2558 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 21:49:59.011000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003776 2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003782 2558 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003785 2558 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003788 2558 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003791 2558 flags.go:64] FLAG: --pod-cidr="" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003794 2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003800 2558 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003803 2558 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003806 2558 flags.go:64] FLAG: --pods-per-core="0" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003809 2558 flags.go:64] FLAG: --port="10250" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003812 2558 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003815 2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02c4c90d066a41650" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003818 2558 flags.go:64] FLAG: --qos-reserved="" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003821 2558 flags.go:64] FLAG: --read-only-port="10255" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003825 2558 flags.go:64] FLAG: --register-node="true" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003828 2558 flags.go:64] FLAG: --register-with-taints="" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003832 2558 flags.go:64] FLAG: --registry-burst="10" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003834 2558 flags.go:64] FLAG: --registry-qps="5" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003837 2558 flags.go:64] FLAG: --reserved-cpus="" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003840 2558 flags.go:64] FLAG: --reserved-memory="" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003843 2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003846 2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003849 2558 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003865 2558 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003869 2558 flags.go:64] FLAG: --runonce="false" Jul 16 21:49:59.011607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003872 2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003875 2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003878 2558 flags.go:64] FLAG: --seccomp-default="false" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003881 2558 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003884 2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003886 2558 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003889 2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003893 2558 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003896 2558 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003899 2558 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003902 2558 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003905 2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003908 2558 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003912 2558 flags.go:64] FLAG: --system-cgroups="" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003915 2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003920 2558 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003923 2558 flags.go:64] FLAG: --tls-cert-file="" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003925 2558 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003929 2558 flags.go:64] FLAG: --tls-min-version="" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003932 2558 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003935 2558 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003939 2558 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003942 2558 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003945 2558 flags.go:64] FLAG: --v="2" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003949 2558 flags.go:64] FLAG: --version="false" Jul 16 21:49:59.012235 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003953 2558 flags.go:64] FLAG: --vmodule="" Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003957 2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.003960 2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004081 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004085 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004088 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004091 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004093 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004096 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004098 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004101 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004105 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004108 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004111 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004114 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004117 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004120 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004123 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004125 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004128 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:59.012849 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004131 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004134 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004136 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004139 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004141 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004143 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004146 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004148 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004152 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004154 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004157 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004159 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004161 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004164 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004166 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004169 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004171 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004173 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004176 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004178 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:59.013696 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004181 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004184 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004186 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004189 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004191 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004193 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004196 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004198 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004201 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004203 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004206 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004208 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004212 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004214 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004217 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004219 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004222 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004224 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004228 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:59.014369 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004231 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004235 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004238 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004240 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004242 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004245 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004247 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004250 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004252 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004255 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004257 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004260 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004263 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004265 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004267 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004270 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004272 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004275 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004277 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004280 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:59.014843 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004282 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004285 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004287 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004290 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004292 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004296 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004298 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004301 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004303 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004306 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004308 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004310 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004313 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004316 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004319 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004322 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004324 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004326 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004329 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004331 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:59.015359 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004334 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004337 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.004341 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.004925 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.011669 2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.011687 2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.012411 2558 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.012421 2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012479 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012484 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012489 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012493 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012496 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012499 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:59.015891 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012502 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012505 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012508 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012511 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012513 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012516 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012519 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012521 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012524 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012527 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012529 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012532 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012534 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012536 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012539 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012541 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012544 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012546 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012549 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012551 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:59.016244 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012554 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012556 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012559 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012562 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012565 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012567 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012570 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012572 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012575 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012577 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012579 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012582 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012584 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012587 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012590 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012592 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012596 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012599 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012601 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012604 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:59.016746 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012606 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012609 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012612 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012614 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012616 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012619 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012621 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012624 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012627 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012629 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012632 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012634 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012637 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012639 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012642 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012644 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012647 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012649 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012652 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012654 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:59.017247 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012657 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012659 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012661 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012664 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012666 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012670 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012673 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012681 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012684 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012687 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012689 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012692 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012694 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012697 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012699 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012703 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012705 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012708 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:59.017727 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012710 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012713 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012715 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012718 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012721 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012724 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012726 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012729 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012731 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012734 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012736 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012739 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012741 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012744 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012746 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:59.018172 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.012751 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012873 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012877 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012880 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012883 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012886 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012889 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012938 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.012996 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013001 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013004 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013007 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013010 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013013 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013015 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013018 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013021 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013023 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013025 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013028 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:59.018546 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013031 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013037 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013040 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013044 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013049 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013053 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013059 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013063 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013067 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013356 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013793 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013799 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013805 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013810 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013815 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013820 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013824 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013829 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013833 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013837 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:59.019041 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013842 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013846 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013879 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013883 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013887 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013896 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013901 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013905 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013909 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013914 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013918 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013925 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013930 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013935 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013945 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013950 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013955 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013959 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013963 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:59.019547 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013968 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013973 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013977 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013982 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013988 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013992 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.013996 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014004 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014009 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014013 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014019 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014027 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014032 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014035 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014038 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014041 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014044 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014046 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014049 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:59.020153 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014052 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014057 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014061 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014065 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014070 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014078 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014082 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014086 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014090 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014094 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014099 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014104 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014108 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014113 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014117 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014121 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014127 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014136 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014141 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014145 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:59.020603 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014149 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:59.021096 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:49:59.014153 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:59.021096 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.014162 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:49:59.021096 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.014596 2558 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 21:49:59.021096 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.017963 2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 21:49:59.021096 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.019305 2558 server.go:1019] "Starting client certificate rotation" Jul 16 21:49:59.021096 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.019409 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 21:49:59.021096 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.019441 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 21:49:59.021756 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.021742 2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 21:49:59.026437 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.026418 2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 21:49:59.038947 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.038912 2558 log.go:25] "Validated CRI v1 runtime API" Jul 16 21:49:59.042719 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.042700 2558 log.go:25] "Validated CRI v1 image API" Jul 16 21:49:59.044228 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.044209 2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 21:49:59.046471 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.046452 2558 fs.go:135] Filesystem UUIDs: map[68fa4ef7-27a5-4516-968a-91a5f6d7261e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 da23a235-aa32-4a7c-b8b0-d0ea26809d49:/dev/nvme0n1p4] Jul 16 21:49:59.046529 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.046472 2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 21:49:59.046793 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.046776 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 21:49:59.050847 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.050737 2558 manager.go:216] Machine: {Timestamp:2026-07-16 21:49:59.050265323 +0000 UTC m=+0.310175758 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122163 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec223eaa150c77a86843bbb305864535 SystemUUID:ec223eaa-150c-77a8-6843-bbb305864535 BootID:e0c59682-13a2-4ad7-9a85-b4b64d0c1cf8 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:12:f2:5d:4a:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:12:f2:5d:4a:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:b0:67:38:24:90 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 21:49:59.050847 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.050841 2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 21:49:59.050989 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.050935 2558 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 21:49:59.053973 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.053943 2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 21:49:59.054139 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.053975 2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-231.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 21:49:59.054719 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.054707 2558 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 21:49:59.054773 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.054726 2558 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 21:49:59.054773 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.054738 2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 21:49:59.054773 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.054755 2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 21:49:59.055891 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.055879 2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 21:49:59.058979 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.058965 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 16 21:49:59.059111 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.059100 2558 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 21:49:59.059171 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.059139 2558 kubelet.go:486] "Attempting to sync node with API server" Jul 16 21:49:59.059171 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.059153 2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 21:49:59.059171 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.059168 2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 21:49:59.059287 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.059181 2558 kubelet.go:398] "Adding apiserver pod source" Jul 16 21:49:59.059287 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.059194 2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 21:49:59.059607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.059583 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 21:49:59.059607 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.059608 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 21:49:59.061696 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.061677 2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 21:49:59.066305 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.066286 2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 21:49:59.066373 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.066320 2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 21:49:59.066970 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.066957 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 21:49:59.067015 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.066974 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 21:49:59.067015 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.066980 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 21:49:59.067015 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.066986 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 21:49:59.067015 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.066994 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 21:49:59.067015 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.067000 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 21:49:59.067142 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.067030 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 21:49:59.067142 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.067036 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 21:49:59.067142 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.067043 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 21:49:59.067142 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.067049 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 21:49:59.067142 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.067065 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 21:49:59.067142 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.067072 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 21:49:59.067841 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.067831 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 21:49:59.067841 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.067841 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 21:49:59.069074 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.069052 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lnzdf" Jul 16 21:49:59.070291 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.070266 2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-231.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 21:49:59.070930 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.070903 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-231.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 21:49:59.071003 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.070903 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 21:49:59.071003 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.070986 2558 server.go:1268] "Started kubelet" Jul 16 21:49:59.071100 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.071056 2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 21:49:59.071164 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.071107 2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 21:49:59.071206 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.071182 2558 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 21:49:59.072251 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.072235 2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 21:49:59.072261 ip-10-0-131-231 systemd[1]: Started Kubernetes Kubelet. Jul 16 21:49:59.073001 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.072927 2558 server.go:310] "Adding debug handlers to kubelet server" Jul 16 21:49:59.076413 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.076381 2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 21:49:59.078248 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.078234 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lnzdf" Jul 16 21:49:59.078463 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.077535 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-231.ec2.internal.18c2e38d742a31a9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-231.ec2.internal,UID:ip-10-0-131-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-231.ec2.internal,},FirstTimestamp:2026-07-16 21:49:59.070953897 +0000 UTC m=+0.330864315,LastTimestamp:2026-07-16 21:49:59.070953897 +0000 UTC m=+0.330864315,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-231.ec2.internal,}" Jul 16 21:49:59.080119 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.080103 2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 21:49:59.080119 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.080112 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 21:49:59.080596 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.080209 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:49:59.080780 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.080767 2558 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 21:49:59.080780 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.080783 2558 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 21:49:59.080939 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.080928 2558 reconstruct.go:97] "Volume reconstruction finished" Jul 16 21:49:59.080983 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.080940 2558 reconciler.go:29] "Reconciler: start to sync state" Jul 16 21:49:59.080983 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.080938 2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 21:49:59.080983 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.080951 2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 21:49:59.080983 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.080960 2558 factory.go:55] Registering systemd factory Jul 16 21:49:59.080983 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.080970 2558 factory.go:223] Registration of the systemd container factory successfully Jul 16 21:49:59.081191 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.081183 2558 factory.go:153] Registering CRI-O factory Jul 16 21:49:59.081253 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.081194 2558 factory.go:223] Registration of the crio container factory successfully Jul 16 21:49:59.081253 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.081211 2558 factory.go:103] Registering Raw factory Jul 16 21:49:59.081253 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.081222 2558 manager.go:1196] Started watching for new ooms in manager Jul 16 21:49:59.081568 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.081558 2558 manager.go:349] Starting recovery of all containers Jul 16 21:49:59.086708 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.086646 2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-231.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 21:49:59.086814 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.086751 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 21:49:59.089886 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.089819 2558 manager.go:354] Recovery completed Jul 16 21:49:59.098776 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.098761 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:59.100478 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.100459 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:59.100566 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.100495 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:59.100566 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.100509 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:59.101070 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.101055 2558 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 21:49:59.101070 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.101068 2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 21:49:59.101157 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.101083 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 16 21:49:59.101190 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.101059 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:59.101951 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.101935 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:59.102014 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.101964 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:59.102014 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.101974 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:59.102252 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.102240 2558 policy_none.go:49] "None policy: Start" Jul 16 21:49:59.102284 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.102257 2558 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 21:49:59.102284 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.102267 2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 21:49:59.103340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.103325 2558 policy_none.go:47] "Start" Jul 16 21:49:59.160051 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.159763 2558 manager.go:337] "Starting Device Plugin manager" Jul 16 21:49:59.171874 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.160075 2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 21:49:59.171874 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.160106 2558 server.go:87] "Starting device plugin registration server" Jul 16 21:49:59.171874 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.160363 2558 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 21:49:59.171874 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.160373 2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 21:49:59.171874 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.160490 2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 21:49:59.171874 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.160592 2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 21:49:59.171874 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.160604 2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 21:49:59.171874 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.161312 2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 21:49:59.171874 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.161364 2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:49:59.254072 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.253981 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 21:49:59.255364 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.255342 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 21:49:59.255435 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.255397 2558 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 21:49:59.255435 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.255426 2558 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 21:49:59.255537 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.255467 2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 21:49:59.257711 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.257682 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:49:59.260466 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.260450 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:59.261289 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.261276 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:59.261361 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.261305 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:59.261361 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.261315 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:59.261361 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.261338 2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.267118 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.267100 2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.267203 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.267124 2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-231.ec2.internal\": node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:49:59.283816 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.283793 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:49:59.356503 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.356465 2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-231.ec2.internal"] Jul 16 21:49:59.356660 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.356569 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:59.357570 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.357554 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:59.357633 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.357587 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:59.357633 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.357600 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:59.358882 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.358870 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:59.358970 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.358955 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.359039 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.358985 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:59.359584 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.359561 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:59.359702 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.359594 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:59.359702 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.359606 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:59.359702 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.359613 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:59.359702 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.359630 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:59.359702 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.359638 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:59.360755 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.360740 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.360833 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.360767 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:59.361456 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.361440 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:59.361545 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.361468 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:59.361545 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.361480 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:59.382885 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.382842 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73ccc2381eb0957c4c8d3c3bbac73808-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal\" (UID: \"73ccc2381eb0957c4c8d3c3bbac73808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.382996 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.382886 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73ccc2381eb0957c4c8d3c3bbac73808-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal\" (UID: \"73ccc2381eb0957c4c8d3c3bbac73808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.382996 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.382906 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f66b8fb68e2cffebe18d00c9a2172a9c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-231.ec2.internal\" (UID: \"f66b8fb68e2cffebe18d00c9a2172a9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.383898 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.383881 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:49:59.390512 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.390495 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-231.ec2.internal\" not found" node="ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.394766 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.394748 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-231.ec2.internal\" not found" node="ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.483421 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.483395 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73ccc2381eb0957c4c8d3c3bbac73808-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal\" (UID: \"73ccc2381eb0957c4c8d3c3bbac73808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.483421 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.483425 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73ccc2381eb0957c4c8d3c3bbac73808-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal\" (UID: \"73ccc2381eb0957c4c8d3c3bbac73808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.483631 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.483444 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f66b8fb68e2cffebe18d00c9a2172a9c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-231.ec2.internal\" (UID: \"f66b8fb68e2cffebe18d00c9a2172a9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.483631 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.483492 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f66b8fb68e2cffebe18d00c9a2172a9c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-231.ec2.internal\" (UID: \"f66b8fb68e2cffebe18d00c9a2172a9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.483631 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.483500 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73ccc2381eb0957c4c8d3c3bbac73808-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal\" (UID: \"73ccc2381eb0957c4c8d3c3bbac73808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.483631 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.483505 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73ccc2381eb0957c4c8d3c3bbac73808-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal\" (UID: \"73ccc2381eb0957c4c8d3c3bbac73808\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.484459 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.484440 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:49:59.585105 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.585027 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:49:59.685462 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.685428 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:49:59.692609 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.692583 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.698109 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:49:59.698093 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.ec2.internal" Jul 16 21:49:59.785656 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.785619 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:49:59.886255 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.886176 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:49:59.986687 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:49:59.986650 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:50:00.018841 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.018812 2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 21:50:00.019297 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.018974 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 21:50:00.086924 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:00.086894 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:50:00.090513 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.090486 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:44:59 +0000 UTC" deadline="2028-03-02 16:16:03.938959742 +0000 UTC" Jul 16 21:50:00.090513 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.090511 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14274h26m3.848451527s" Jul 16 21:50:00.128747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.128696 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:50:00.187403 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:00.187336 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:50:00.191424 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.191404 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:50:00.218362 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:50:00.218319 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66b8fb68e2cffebe18d00c9a2172a9c.slice/crio-b4cc9c5250ffcec261ac5608e577a8e8b5dec80d01e3fb515681377b9c0a3f28 WatchSource:0}: Error finding container b4cc9c5250ffcec261ac5608e577a8e8b5dec80d01e3fb515681377b9c0a3f28: Status 404 returned error can't find the container with id b4cc9c5250ffcec261ac5608e577a8e8b5dec80d01e3fb515681377b9c0a3f28 Jul 16 21:50:00.218593 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:50:00.218570 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ccc2381eb0957c4c8d3c3bbac73808.slice/crio-fd77d2c068736b348a5bea7b575e9d7fb8cddbbfcceb3e9b17da3f6f3f9ea060 WatchSource:0}: Error finding container fd77d2c068736b348a5bea7b575e9d7fb8cddbbfcceb3e9b17da3f6f3f9ea060: Status 404 returned error can't find the container with id fd77d2c068736b348a5bea7b575e9d7fb8cddbbfcceb3e9b17da3f6f3f9ea060 Jul 16 21:50:00.225012 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.224996 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:50:00.259890 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.259822 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.ec2.internal" event={"ID":"f66b8fb68e2cffebe18d00c9a2172a9c","Type":"ContainerStarted","Data":"b4cc9c5250ffcec261ac5608e577a8e8b5dec80d01e3fb515681377b9c0a3f28"} Jul 16 21:50:00.260796 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.260776 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" event={"ID":"73ccc2381eb0957c4c8d3c3bbac73808","Type":"ContainerStarted","Data":"fd77d2c068736b348a5bea7b575e9d7fb8cddbbfcceb3e9b17da3f6f3f9ea060"} Jul 16 21:50:00.288293 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:00.288262 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-231.ec2.internal\" not found" Jul 16 21:50:00.321135 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.321111 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:50:00.380344 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.380315 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" Jul 16 21:50:00.420884 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.420841 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 21:50:00.421944 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.421931 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.ec2.internal" Jul 16 21:50:00.448415 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:00.448368 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 21:50:01.060656 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.060574 2558 apiserver.go:52] "Watching apiserver" Jul 16 21:50:01.071429 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.071398 2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 21:50:01.074442 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.074405 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc","openshift-cluster-node-tuning-operator/tuned-sx6fl","openshift-dns/node-resolver-b2wz8","openshift-image-registry/node-ca-8stlx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal","openshift-multus/multus-additional-cni-plugins-5x7sg","openshift-multus/multus-bzncd","openshift-network-diagnostics/network-check-target-nhjhp","kube-system/global-pull-secret-syncer-jkdtg","kube-system/konnectivity-agent-dmp8k","kube-system/kube-apiserver-proxy-ip-10-0-131-231.ec2.internal","openshift-multus/network-metrics-daemon-wkshg","openshift-network-operator/iptables-alerter-nhs4l","openshift-ovn-kubernetes/ovnkube-node-7ttlc"] Jul 16 21:50:01.076781 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.076107 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.077923 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.077898 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:01.078031 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.077980 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:01.082870 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.079833 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 21:50:01.082870 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.079872 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 21:50:01.082870 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.080013 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 21:50:01.082870 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.080123 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wvrrk\"" Jul 16 21:50:01.082870 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.080184 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 21:50:01.082870 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.080367 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b2wz8" Jul 16 21:50:01.082870 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.080733 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 21:50:01.082870 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.082479 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8stlx" Jul 16 21:50:01.083276 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.083086 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 21:50:01.083276 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.083258 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 21:50:01.083339 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.083316 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vcdm4\"" Jul 16 21:50:01.083961 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.083938 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.085535 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.085515 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 21:50:01.085773 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.085752 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w5sqs\"" Jul 16 21:50:01.085986 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.085968 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 21:50:01.086484 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.086293 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 21:50:01.086484 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.086477 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 21:50:01.086628 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.086614 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 21:50:01.086911 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.086883 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 21:50:01.087195 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.087179 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.087291 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.087255 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6d4mc\"" Jul 16 21:50:01.087697 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.087675 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:01.087774 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.087744 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:01.089598 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.089578 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 21:50:01.089839 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.089824 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2x99\"" Jul 16 21:50:01.090085 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.090068 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nhs4l" Jul 16 21:50:01.090163 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.090127 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 21:50:01.090903 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.090884 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-system-cni-dir\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.090993 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.090916 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/028411f0-adf7-453e-bb59-1b27079457e4-cni-binary-copy\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.090993 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.090962 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-hostroot\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.091113 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091007 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3347b12c-12e0-4f72-a9eb-9480db6fd042-host\") pod \"node-ca-8stlx\" (UID: \"3347b12c-12e0-4f72-a9eb-9480db6fd042\") " pod="openshift-image-registry/node-ca-8stlx" Jul 16 21:50:01.091113 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091032 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-device-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.091113 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091059 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-var-lib-cni-multus\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.091546 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091076 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-multus-socket-dir-parent\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.091626 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091573 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-var-lib-kubelet\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.091626 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091603 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prp4g\" (UniqueName: \"kubernetes.io/projected/c29a9f8e-4098-45a1-8c53-84c0fc20f0c1-kube-api-access-prp4g\") pod \"node-resolver-b2wz8\" (UID: \"c29a9f8e-4098-45a1-8c53-84c0fc20f0c1\") " pod="openshift-dns/node-resolver-b2wz8" Jul 16 21:50:01.091737 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091627 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.091737 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091322 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.091737 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091651 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-run-k8s-cni-cncf-io\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.091737 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091675 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-var-lib-cni-bin\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.091737 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091703 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-etc-kubernetes\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.092027 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091746 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-run-netns\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.092027 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091788 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c29a9f8e-4098-45a1-8c53-84c0fc20f0c1-hosts-file\") pod \"node-resolver-b2wz8\" (UID: \"c29a9f8e-4098-45a1-8c53-84c0fc20f0c1\") " pod="openshift-dns/node-resolver-b2wz8" Jul 16 21:50:01.092027 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091822 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29a9f8e-4098-45a1-8c53-84c0fc20f0c1-tmp-dir\") pod \"node-resolver-b2wz8\" (UID: \"c29a9f8e-4098-45a1-8c53-84c0fc20f0c1\") " pod="openshift-dns/node-resolver-b2wz8" Jul 16 21:50:01.092027 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091858 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-cnibin\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.092027 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091918 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-os-release\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.092027 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091939 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-run-multus-certs\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.092027 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.091972 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnzvm\" (UniqueName: \"kubernetes.io/projected/028411f0-adf7-453e-bb59-1b27079457e4-kube-api-access-wnzvm\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.092027 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092006 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-socket-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.092430 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092033 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2czt\" (UniqueName: \"kubernetes.io/projected/b8d78eee-2888-4499-a24b-c2bc1123a5a2-kube-api-access-g2czt\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.092430 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092071 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-multus-cni-dir\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.092430 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092104 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-multus-conf-dir\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.092430 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092134 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.092430 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092180 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-registration-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.092430 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092229 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/028411f0-adf7-453e-bb59-1b27079457e4-multus-daemon-config\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.092430 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092263 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4kv\" (UniqueName: \"kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv\") pod \"network-check-target-nhjhp\" (UID: \"21d8755c-3957-4432-87d7-312fb00a1d6a\") " pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:01.092430 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092305 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3347b12c-12e0-4f72-a9eb-9480db6fd042-serviceca\") pod \"node-ca-8stlx\" (UID: \"3347b12c-12e0-4f72-a9eb-9480db6fd042\") " pod="openshift-image-registry/node-ca-8stlx" Jul 16 21:50:01.092430 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092330 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9j7k\" (UniqueName: \"kubernetes.io/projected/3347b12c-12e0-4f72-a9eb-9480db6fd042-kube-api-access-z9j7k\") pod \"node-ca-8stlx\" (UID: \"3347b12c-12e0-4f72-a9eb-9480db6fd042\") " pod="openshift-image-registry/node-ca-8stlx" Jul 16 21:50:01.092430 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092355 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-sys-fs\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.092925 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092497 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:50:01.092925 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092586 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 21:50:01.092925 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092755 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:01.092925 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092766 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 21:50:01.092925 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.092856 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7mtkd\"" Jul 16 21:50:01.094234 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.094214 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p24gz\"" Jul 16 21:50:01.094613 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.094597 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.095743 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.095723 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:50:01.096550 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.096529 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:01.096632 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.096606 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:01.100862 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.100813 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 21:50:01.100862 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.100823 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 21:50:01.101587 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.101567 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 21:50:01.101785 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.101770 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 21:50:01.102027 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.102001 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 21:50:01.102122 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.102108 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 21:50:01.105664 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.105593 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 21:50:01.105664 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.105600 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l262l\"" Jul 16 21:50:01.106150 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.106007 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 21:50:01.106150 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.106017 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 21:50:01.106150 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.106022 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 21:50:01.106542 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.106525 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-958rh\"" Jul 16 21:50:01.156020 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.155994 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dfndx" Jul 16 21:50:01.167772 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.167750 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dfndx" Jul 16 21:50:01.183061 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.183022 2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 21:50:01.192582 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.192558 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-run-ovn\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.192700 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.192597 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/028411f0-adf7-453e-bb59-1b27079457e4-cni-binary-copy\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.192700 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.192623 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-hostroot\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.192816 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.192775 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-hostroot\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.193216 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193186 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/028411f0-adf7-453e-bb59-1b27079457e4-cni-binary-copy\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.193390 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193372 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3347b12c-12e0-4f72-a9eb-9480db6fd042-host\") pod \"node-ca-8stlx\" (UID: \"3347b12c-12e0-4f72-a9eb-9480db6fd042\") " pod="openshift-image-registry/node-ca-8stlx" Jul 16 21:50:01.193469 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193408 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-device-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193629 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193667 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193696 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca08e685-21d1-4a2a-8c21-5579a3ccb8dc-host-slash\") pod \"iptables-alerter-nhs4l\" (UID: \"ca08e685-21d1-4a2a-8c21-5579a3ccb8dc\") " pod="openshift-network-operator/iptables-alerter-nhs4l" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193740 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-tmp\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193770 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-var-lib-cni-multus\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193794 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-cnibin\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193817 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdq89\" (UniqueName: \"kubernetes.io/projected/cd481684-0c85-4cbe-ab71-3e6130ab1131-kube-api-access-fdq89\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193841 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-sysctl-d\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193866 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193913 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-ovnkube-script-lib\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193940 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-multus-socket-dir-parent\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193965 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/079499fc-0421-434a-8a61-d0d075b89be1-konnectivity-ca\") pod \"konnectivity-agent-dmp8k\" (UID: \"079499fc-0421-434a-8a61-d0d075b89be1\") " pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193989 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-var-lib-cni-bin\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193669 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-device-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194016 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-etc-kubernetes\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.193706 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3347b12c-12e0-4f72-a9eb-9480db6fd042-host\") pod \"node-ca-8stlx\" (UID: \"3347b12c-12e0-4f72-a9eb-9480db6fd042\") " pod="openshift-image-registry/node-ca-8stlx" Jul 16 21:50:01.194165 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194058 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-run\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194060 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-etc-kubernetes\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194124 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-var-lib-cni-bin\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194153 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-var-lib-cni-multus\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194217 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-multus-socket-dir-parent\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194252 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-tuned\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194281 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c29a9f8e-4098-45a1-8c53-84c0fc20f0c1-hosts-file\") pod \"node-resolver-b2wz8\" (UID: \"c29a9f8e-4098-45a1-8c53-84c0fc20f0c1\") " pod="openshift-dns/node-resolver-b2wz8" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194306 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-system-cni-dir\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194334 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd481684-0c85-4cbe-ab71-3e6130ab1131-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194360 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c1ed479-ddc7-4099-b273-ddd8c98534b4-kubelet-config\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194414 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-host\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194441 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-run-netns\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194476 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194504 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-ovnkube-config\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194533 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-os-release\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194564 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-run-multus-certs\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194590 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnzvm\" (UniqueName: \"kubernetes.io/projected/028411f0-adf7-453e-bb59-1b27079457e4-kube-api-access-wnzvm\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.195147 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194627 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-socket-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194653 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/079499fc-0421-434a-8a61-d0d075b89be1-agent-certs\") pod \"konnectivity-agent-dmp8k\" (UID: \"079499fc-0421-434a-8a61-d0d075b89be1\") " pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194680 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxk62\" (UniqueName: \"kubernetes.io/projected/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-kube-api-access-nxk62\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194707 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-log-socket\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194738 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-cni-bin\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194765 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca08e685-21d1-4a2a-8c21-5579a3ccb8dc-iptables-alerter-script\") pod \"iptables-alerter-nhs4l\" (UID: \"ca08e685-21d1-4a2a-8c21-5579a3ccb8dc\") " pod="openshift-network-operator/iptables-alerter-nhs4l" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194798 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-lib-modules\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194824 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-var-lib-kubelet\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194848 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-slash\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194871 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-node-log\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194895 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-ovn-node-metrics-cert\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194953 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.194985 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/028411f0-adf7-453e-bb59-1b27079457e4-multus-daemon-config\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195011 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4kv\" (UniqueName: \"kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv\") pod \"network-check-target-nhjhp\" (UID: \"21d8755c-3957-4432-87d7-312fb00a1d6a\") " pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195039 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9j7k\" (UniqueName: \"kubernetes.io/projected/3347b12c-12e0-4f72-a9eb-9480db6fd042-kube-api-access-z9j7k\") pod \"node-ca-8stlx\" (UID: \"3347b12c-12e0-4f72-a9eb-9480db6fd042\") " pod="openshift-image-registry/node-ca-8stlx" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195065 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-sysctl-conf\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.196053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195091 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-system-cni-dir\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195119 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnnt\" (UniqueName: \"kubernetes.io/projected/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-kube-api-access-ltnnt\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195144 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-systemd\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195168 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-etc-openvswitch\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195197 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-cni-netd\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195225 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwrv2\" (UniqueName: \"kubernetes.io/projected/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-kube-api-access-cwrv2\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195262 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-var-lib-kubelet\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195289 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prp4g\" (UniqueName: \"kubernetes.io/projected/c29a9f8e-4098-45a1-8c53-84c0fc20f0c1-kube-api-access-prp4g\") pod \"node-resolver-b2wz8\" (UID: \"c29a9f8e-4098-45a1-8c53-84c0fc20f0c1\") " pod="openshift-dns/node-resolver-b2wz8" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195315 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195346 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-modprobe-d\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195375 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-run-openvswitch\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195402 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-run-k8s-cni-cncf-io\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195431 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-os-release\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195463 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-kubernetes\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195495 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59krx\" (UniqueName: \"kubernetes.io/projected/ca08e685-21d1-4a2a-8c21-5579a3ccb8dc-kube-api-access-59krx\") pod \"iptables-alerter-nhs4l\" (UID: \"ca08e685-21d1-4a2a-8c21-5579a3ccb8dc\") " pod="openshift-network-operator/iptables-alerter-nhs4l" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195521 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-sysconfig\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195549 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-run-netns\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.196844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195578 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29a9f8e-4098-45a1-8c53-84c0fc20f0c1-tmp-dir\") pod \"node-resolver-b2wz8\" (UID: \"c29a9f8e-4098-45a1-8c53-84c0fc20f0c1\") " pod="openshift-dns/node-resolver-b2wz8" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195605 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd481684-0c85-4cbe-ab71-3e6130ab1131-cni-binary-copy\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195632 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-kubelet\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195668 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-cnibin\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195694 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2czt\" (UniqueName: \"kubernetes.io/projected/b8d78eee-2888-4499-a24b-c2bc1123a5a2-kube-api-access-g2czt\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195723 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-sys\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195753 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-systemd-units\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195780 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-var-lib-openvswitch\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195807 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-env-overrides\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195834 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-multus-cni-dir\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195861 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-multus-conf-dir\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195891 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195921 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-registration-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195970 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3347b12c-12e0-4f72-a9eb-9480db6fd042-serviceca\") pod \"node-ca-8stlx\" (UID: \"3347b12c-12e0-4f72-a9eb-9480db6fd042\") " pod="openshift-image-registry/node-ca-8stlx" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.195996 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-sys-fs\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.196024 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd481684-0c85-4cbe-ab71-3e6130ab1131-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.196051 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c1ed479-ddc7-4099-b273-ddd8c98534b4-dbus\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:01.197698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.196076 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-run-systemd\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.198606 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.196197 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c29a9f8e-4098-45a1-8c53-84c0fc20f0c1-hosts-file\") pod \"node-resolver-b2wz8\" (UID: \"c29a9f8e-4098-45a1-8c53-84c0fc20f0c1\") " pod="openshift-dns/node-resolver-b2wz8" Jul 16 21:50:01.198606 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.196346 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-os-release\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.198606 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.196394 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-run-multus-certs\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.198606 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.196786 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-socket-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.198606 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.197474 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/028411f0-adf7-453e-bb59-1b27079457e4-multus-daemon-config\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.198606 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.198034 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-system-cni-dir\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.198606 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.198140 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-var-lib-kubelet\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.198606 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.198451 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-etc-selinux\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.198606 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.198526 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-run-k8s-cni-cncf-io\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.199083 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.198612 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-host-run-netns\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.199083 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.198965 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29a9f8e-4098-45a1-8c53-84c0fc20f0c1-tmp-dir\") pod \"node-resolver-b2wz8\" (UID: \"c29a9f8e-4098-45a1-8c53-84c0fc20f0c1\") " pod="openshift-dns/node-resolver-b2wz8" Jul 16 21:50:01.199184 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.199081 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-cnibin\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.199315 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.199288 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-multus-conf-dir\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.199400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.199356 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.199443 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.199431 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-registration-dir\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.199517 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.199499 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/028411f0-adf7-453e-bb59-1b27079457e4-multus-cni-dir\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.199595 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.199579 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b8d78eee-2888-4499-a24b-c2bc1123a5a2-sys-fs\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.199832 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.199817 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3347b12c-12e0-4f72-a9eb-9480db6fd042-serviceca\") pod \"node-ca-8stlx\" (UID: \"3347b12c-12e0-4f72-a9eb-9480db6fd042\") " pod="openshift-image-registry/node-ca-8stlx" Jul 16 21:50:01.205515 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.205483 2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 21:50:01.208874 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.208845 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:50:01.208992 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.208895 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:50:01.208992 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.208909 2558 projected.go:196] Error preparing data for projected volume kube-api-access-tw4kv for pod openshift-network-diagnostics/network-check-target-nhjhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:01.208992 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.208986 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv podName:21d8755c-3957-4432-87d7-312fb00a1d6a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:01.708965539 +0000 UTC m=+2.968875964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tw4kv" (UniqueName: "kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv") pod "network-check-target-nhjhp" (UID: "21d8755c-3957-4432-87d7-312fb00a1d6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:01.209367 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.209341 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnzvm\" (UniqueName: \"kubernetes.io/projected/028411f0-adf7-453e-bb59-1b27079457e4-kube-api-access-wnzvm\") pod \"multus-bzncd\" (UID: \"028411f0-adf7-453e-bb59-1b27079457e4\") " pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.210453 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.210428 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prp4g\" (UniqueName: \"kubernetes.io/projected/c29a9f8e-4098-45a1-8c53-84c0fc20f0c1-kube-api-access-prp4g\") pod \"node-resolver-b2wz8\" (UID: \"c29a9f8e-4098-45a1-8c53-84c0fc20f0c1\") " pod="openshift-dns/node-resolver-b2wz8" Jul 16 21:50:01.210714 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.210691 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2czt\" (UniqueName: \"kubernetes.io/projected/b8d78eee-2888-4499-a24b-c2bc1123a5a2-kube-api-access-g2czt\") pod \"aws-ebs-csi-driver-node-d9glc\" (UID: \"b8d78eee-2888-4499-a24b-c2bc1123a5a2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.211621 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.211594 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9j7k\" (UniqueName: \"kubernetes.io/projected/3347b12c-12e0-4f72-a9eb-9480db6fd042-kube-api-access-z9j7k\") pod \"node-ca-8stlx\" (UID: \"3347b12c-12e0-4f72-a9eb-9480db6fd042\") " pod="openshift-image-registry/node-ca-8stlx" Jul 16 21:50:01.297004 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.296967 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-modprobe-d\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.297004 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297005 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-run-openvswitch\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297024 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-os-release\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297051 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-kubernetes\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297071 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59krx\" (UniqueName: \"kubernetes.io/projected/ca08e685-21d1-4a2a-8c21-5579a3ccb8dc-kube-api-access-59krx\") pod \"iptables-alerter-nhs4l\" (UID: \"ca08e685-21d1-4a2a-8c21-5579a3ccb8dc\") " pod="openshift-network-operator/iptables-alerter-nhs4l" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297087 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-sysconfig\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297084 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-run-openvswitch\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297111 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd481684-0c85-4cbe-ab71-3e6130ab1131-cni-binary-copy\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297134 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-kubelet\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297154 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-modprobe-d\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297160 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-sys\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297180 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-os-release\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297166 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-kubernetes\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297206 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-sys\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297209 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-systemd-units\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297239 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-sysconfig\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.297254 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297244 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-kubelet\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297268 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-var-lib-openvswitch\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297294 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-var-lib-openvswitch\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297273 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-systemd-units\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297323 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-env-overrides\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297362 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd481684-0c85-4cbe-ab71-3e6130ab1131-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297400 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c1ed479-ddc7-4099-b273-ddd8c98534b4-dbus\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297429 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-run-systemd\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297457 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-run-ovn\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297489 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c1ed479-ddc7-4099-b273-ddd8c98534b4-dbus\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297492 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297523 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297558 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca08e685-21d1-4a2a-8c21-5579a3ccb8dc-host-slash\") pod \"iptables-alerter-nhs4l\" (UID: \"ca08e685-21d1-4a2a-8c21-5579a3ccb8dc\") " pod="openshift-network-operator/iptables-alerter-nhs4l" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297585 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-tmp\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297619 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-cnibin\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297792 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-env-overrides\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297822 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd481684-0c85-4cbe-ab71-3e6130ab1131-cni-binary-copy\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297884 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-run-ovn\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.297964 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.297922 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.297940 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd481684-0c85-4cbe-ab71-3e6130ab1131-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298034 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-run-systemd\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298123 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-cnibin\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298139 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298201 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdq89\" (UniqueName: \"kubernetes.io/projected/cd481684-0c85-4cbe-ab71-3e6130ab1131-kube-api-access-fdq89\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298210 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca08e685-21d1-4a2a-8c21-5579a3ccb8dc-host-slash\") pod \"iptables-alerter-nhs4l\" (UID: \"ca08e685-21d1-4a2a-8c21-5579a3ccb8dc\") " pod="openshift-network-operator/iptables-alerter-nhs4l" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298236 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-sysctl-d\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.298269 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret podName:9c1ed479-ddc7-4099-b273-ddd8c98534b4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:01.798239411 +0000 UTC m=+3.058149818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret") pod "global-pull-secret-syncer-jkdtg" (UID: "9c1ed479-ddc7-4099-b273-ddd8c98534b4") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298336 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298361 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-sysctl-d\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298365 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-ovnkube-script-lib\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298426 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/079499fc-0421-434a-8a61-d0d075b89be1-konnectivity-ca\") pod \"konnectivity-agent-dmp8k\" (UID: \"079499fc-0421-434a-8a61-d0d075b89be1\") " pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298449 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-run\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298465 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-tuned\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298427 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-run-ovn-kubernetes\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.298747 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298495 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-system-cni-dir\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298524 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd481684-0c85-4cbe-ab71-3e6130ab1131-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298550 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c1ed479-ddc7-4099-b273-ddd8c98534b4-kubelet-config\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298578 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-host\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298600 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-run-netns\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298625 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298652 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-ovnkube-config\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298681 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/079499fc-0421-434a-8a61-d0d075b89be1-agent-certs\") pod \"konnectivity-agent-dmp8k\" (UID: \"079499fc-0421-434a-8a61-d0d075b89be1\") " pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298705 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxk62\" (UniqueName: \"kubernetes.io/projected/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-kube-api-access-nxk62\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298730 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-log-socket\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298756 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-cni-bin\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298781 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca08e685-21d1-4a2a-8c21-5579a3ccb8dc-iptables-alerter-script\") pod \"iptables-alerter-nhs4l\" (UID: \"ca08e685-21d1-4a2a-8c21-5579a3ccb8dc\") " pod="openshift-network-operator/iptables-alerter-nhs4l" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298797 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-run\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298848 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd481684-0c85-4cbe-ab71-3e6130ab1131-system-cni-dir\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298855 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-ovnkube-script-lib\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298957 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-lib-modules\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.298806 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-lib-modules\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.299564 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299008 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-var-lib-kubelet\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299037 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-slash\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299055 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-host\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299061 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-node-log\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299094 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-node-log\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299124 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-ovn-node-metrics-cert\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299142 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c1ed479-ddc7-4099-b273-ddd8c98534b4-kubelet-config\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299152 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299189 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-run-netns\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299197 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-sysctl-conf\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299208 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299226 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnnt\" (UniqueName: \"kubernetes.io/projected/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-kube-api-access-ltnnt\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299251 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-systemd\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299276 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-etc-openvswitch\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299303 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-cni-netd\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299330 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwrv2\" (UniqueName: \"kubernetes.io/projected/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-kube-api-access-cwrv2\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299345 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd481684-0c85-4cbe-ab71-3e6130ab1131-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299403 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-cni-bin\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300275 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.299571 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:01.300977 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299592 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-ovnkube-config\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300977 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.299632 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs podName:7e676b3e-e7cf-41d1-ac8c-e2177080ea66 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:01.799611863 +0000 UTC m=+3.059522271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs") pod "network-metrics-daemon-wkshg" (UID: "7e676b3e-e7cf-41d1-ac8c-e2177080ea66") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:01.300977 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299153 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/079499fc-0421-434a-8a61-d0d075b89be1-konnectivity-ca\") pod \"konnectivity-agent-dmp8k\" (UID: \"079499fc-0421-434a-8a61-d0d075b89be1\") " pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:01.300977 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299677 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-var-lib-kubelet\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.300977 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299722 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-slash\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300977 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299772 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-sysctl-conf\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.300977 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299824 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-etc-openvswitch\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300977 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299843 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-log-socket\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300977 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299870 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-host-cni-netd\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.300977 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299927 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca08e685-21d1-4a2a-8c21-5579a3ccb8dc-iptables-alerter-script\") pod \"iptables-alerter-nhs4l\" (UID: \"ca08e685-21d1-4a2a-8c21-5579a3ccb8dc\") " pod="openshift-network-operator/iptables-alerter-nhs4l" Jul 16 21:50:01.300977 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.299929 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-systemd\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.301539 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.300995 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-etc-tuned\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.301539 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.301155 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-tmp\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.301814 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.301790 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-ovn-node-metrics-cert\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.302213 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.302191 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/079499fc-0421-434a-8a61-d0d075b89be1-agent-certs\") pod \"konnectivity-agent-dmp8k\" (UID: \"079499fc-0421-434a-8a61-d0d075b89be1\") " pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:01.312963 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.312898 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59krx\" (UniqueName: \"kubernetes.io/projected/ca08e685-21d1-4a2a-8c21-5579a3ccb8dc-kube-api-access-59krx\") pod \"iptables-alerter-nhs4l\" (UID: \"ca08e685-21d1-4a2a-8c21-5579a3ccb8dc\") " pod="openshift-network-operator/iptables-alerter-nhs4l" Jul 16 21:50:01.320508 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.320482 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwrv2\" (UniqueName: \"kubernetes.io/projected/a7fe9137-5027-40c2-aebf-9c0a64d1fe37-kube-api-access-cwrv2\") pod \"ovnkube-node-7ttlc\" (UID: \"a7fe9137-5027-40c2-aebf-9c0a64d1fe37\") " pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.320707 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.320566 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnnt\" (UniqueName: \"kubernetes.io/projected/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-kube-api-access-ltnnt\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:01.320872 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.320850 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdq89\" (UniqueName: \"kubernetes.io/projected/cd481684-0c85-4cbe-ab71-3e6130ab1131-kube-api-access-fdq89\") pod \"multus-additional-cni-plugins-5x7sg\" (UID: \"cd481684-0c85-4cbe-ab71-3e6130ab1131\") " pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.321097 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.321064 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxk62\" (UniqueName: \"kubernetes.io/projected/e866eb17-d39c-4e40-87b2-dd1ab0d3133d-kube-api-access-nxk62\") pod \"tuned-sx6fl\" (UID: \"e866eb17-d39c-4e40-87b2-dd1ab0d3133d\") " pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.335337 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.335307 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:50:01.391773 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.391720 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bzncd" Jul 16 21:50:01.398977 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.398953 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b2wz8" Jul 16 21:50:01.414200 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.414174 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8stlx" Jul 16 21:50:01.419279 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.419259 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" Jul 16 21:50:01.427031 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.427011 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5x7sg" Jul 16 21:50:01.437975 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.437954 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nhs4l" Jul 16 21:50:01.444810 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.444786 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" Jul 16 21:50:01.453521 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.453502 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:01.459516 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.459498 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:01.803838 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.803741 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:01.803838 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.803795 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:01.803838 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:01.803825 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4kv\" (UniqueName: \"kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv\") pod \"network-check-target-nhjhp\" (UID: \"21d8755c-3957-4432-87d7-312fb00a1d6a\") " pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:01.804093 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.803936 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:01.804093 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.803947 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:50:01.804093 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.803966 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:50:01.804093 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.803979 2558 projected.go:196] Error preparing data for projected volume kube-api-access-tw4kv for pod openshift-network-diagnostics/network-check-target-nhjhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:01.804093 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.804004 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret podName:9c1ed479-ddc7-4099-b273-ddd8c98534b4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:02.803985924 +0000 UTC m=+4.063896332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret") pod "global-pull-secret-syncer-jkdtg" (UID: "9c1ed479-ddc7-4099-b273-ddd8c98534b4") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:01.804093 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.803947 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:01.804093 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.804030 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv podName:21d8755c-3957-4432-87d7-312fb00a1d6a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:02.804018162 +0000 UTC m=+4.063928576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tw4kv" (UniqueName: "kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv") pod "network-check-target-nhjhp" (UID: "21d8755c-3957-4432-87d7-312fb00a1d6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:01.804093 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:01.804056 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs podName:7e676b3e-e7cf-41d1-ac8c-e2177080ea66 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:02.804038171 +0000 UTC m=+4.063948586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs") pod "network-metrics-daemon-wkshg" (UID: "7e676b3e-e7cf-41d1-ac8c-e2177080ea66") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:02.171077 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:02.170961 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:45:01 +0000 UTC" deadline="2028-04-04 13:44:32.56632378 +0000 UTC" Jul 16 21:50:02.171077 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:02.171008 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15063h54m30.395321461s" Jul 16 21:50:02.256107 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:02.256065 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:02.256267 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:02.256184 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:02.256267 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:02.256245 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:02.256390 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:02.256369 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:02.281426 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:50:02.281350 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29a9f8e_4098_45a1_8c53_84c0fc20f0c1.slice/crio-6f8f914efab06d9f69cfad55f1507019fd438f6e04e95a399cdb31396584abb6 WatchSource:0}: Error finding container 6f8f914efab06d9f69cfad55f1507019fd438f6e04e95a399cdb31396584abb6: Status 404 returned error can't find the container with id 6f8f914efab06d9f69cfad55f1507019fd438f6e04e95a399cdb31396584abb6 Jul 16 21:50:02.282628 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:50:02.282600 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode866eb17_d39c_4e40_87b2_dd1ab0d3133d.slice/crio-83c3e555057e97dbd0e83edbc9c0173b102e75f5f1fca93a6c30f049e5492c1d WatchSource:0}: Error finding container 83c3e555057e97dbd0e83edbc9c0173b102e75f5f1fca93a6c30f049e5492c1d: Status 404 returned error can't find the container with id 83c3e555057e97dbd0e83edbc9c0173b102e75f5f1fca93a6c30f049e5492c1d Jul 16 21:50:02.284112 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:50:02.284074 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd481684_0c85_4cbe_ab71_3e6130ab1131.slice/crio-0780b281d7135e61816635dcef3e205f86031de814604865030da268ad49b624 WatchSource:0}: Error finding container 0780b281d7135e61816635dcef3e205f86031de814604865030da268ad49b624: Status 404 returned error can't find the container with id 0780b281d7135e61816635dcef3e205f86031de814604865030da268ad49b624 Jul 16 21:50:02.290887 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:50:02.290650 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079499fc_0421_434a_8a61_d0d075b89be1.slice/crio-5c96d284a34b7fe6a96909b68cdf805ae4062886a7075e894052c094ac018dd5 WatchSource:0}: Error finding container 5c96d284a34b7fe6a96909b68cdf805ae4062886a7075e894052c094ac018dd5: Status 404 returned error can't find the container with id 5c96d284a34b7fe6a96909b68cdf805ae4062886a7075e894052c094ac018dd5 Jul 16 21:50:02.293287 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:50:02.292568 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3347b12c_12e0_4f72_a9eb_9480db6fd042.slice/crio-82ea27551de6d1dd5eae14972e7d608d28f65ce18169fcf5adee14819cfe700f WatchSource:0}: Error finding container 82ea27551de6d1dd5eae14972e7d608d28f65ce18169fcf5adee14819cfe700f: Status 404 returned error can't find the container with id 82ea27551de6d1dd5eae14972e7d608d28f65ce18169fcf5adee14819cfe700f Jul 16 21:50:02.294563 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:50:02.294373 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028411f0_adf7_453e_bb59_1b27079457e4.slice/crio-d5dadbf8802bd66d323d84227d239f04774751f107847c964a814d93f9c897b1 WatchSource:0}: Error finding container d5dadbf8802bd66d323d84227d239f04774751f107847c964a814d93f9c897b1: Status 404 returned error can't find the container with id d5dadbf8802bd66d323d84227d239f04774751f107847c964a814d93f9c897b1 Jul 16 21:50:02.819452 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:02.819162 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:02.819624 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:02.819455 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:02.819624 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:02.819348 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:02.819624 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:02.819559 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret podName:9c1ed479-ddc7-4099-b273-ddd8c98534b4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:04.819538281 +0000 UTC m=+6.079448699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret") pod "global-pull-secret-syncer-jkdtg" (UID: "9c1ed479-ddc7-4099-b273-ddd8c98534b4") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:02.819624 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:02.819578 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:02.819624 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:02.819586 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:50:02.819624 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:02.819611 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:50:02.819624 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:02.819623 2558 projected.go:196] Error preparing data for projected volume kube-api-access-tw4kv for pod openshift-network-diagnostics/network-check-target-nhjhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:02.819974 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:02.819642 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs podName:7e676b3e-e7cf-41d1-ac8c-e2177080ea66 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:04.819624353 +0000 UTC m=+6.079534768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs") pod "network-metrics-daemon-wkshg" (UID: "7e676b3e-e7cf-41d1-ac8c-e2177080ea66") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:02.819974 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:02.819485 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4kv\" (UniqueName: \"kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv\") pod \"network-check-target-nhjhp\" (UID: \"21d8755c-3957-4432-87d7-312fb00a1d6a\") " pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:02.819974 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:02.819661 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv podName:21d8755c-3957-4432-87d7-312fb00a1d6a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:04.819650576 +0000 UTC m=+6.079560985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tw4kv" (UniqueName: "kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv") pod "network-check-target-nhjhp" (UID: "21d8755c-3957-4432-87d7-312fb00a1d6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:03.171711 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.171558 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:45:01 +0000 UTC" deadline="2028-02-15 05:36:02.325945291 +0000 UTC" Jul 16 21:50:03.171711 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.171628 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13879h45m59.154322423s" Jul 16 21:50:03.258897 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.258862 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:03.259085 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:03.259023 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:03.273693 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.273526 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" event={"ID":"b8d78eee-2888-4499-a24b-c2bc1123a5a2","Type":"ContainerStarted","Data":"4bfb4e4ae580b9010983e03679c1138029e34d598ccf087863dc9a96371a74e0"} Jul 16 21:50:03.276035 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.275978 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" event={"ID":"a7fe9137-5027-40c2-aebf-9c0a64d1fe37","Type":"ContainerStarted","Data":"dd9eb10ad0a2a1496499a526e1fb598a723cdbe5e5219760b241c0f88afa7dff"} Jul 16 21:50:03.283997 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.283916 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" event={"ID":"e866eb17-d39c-4e40-87b2-dd1ab0d3133d","Type":"ContainerStarted","Data":"83c3e555057e97dbd0e83edbc9c0173b102e75f5f1fca93a6c30f049e5492c1d"} Jul 16 21:50:03.313783 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.313746 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.ec2.internal" event={"ID":"f66b8fb68e2cffebe18d00c9a2172a9c","Type":"ContainerStarted","Data":"7a1d49fc16080e4001fe193fd1be781f473bfd0848223ea647609df1e55ee8ea"} Jul 16 21:50:03.322140 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.322081 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nhs4l" event={"ID":"ca08e685-21d1-4a2a-8c21-5579a3ccb8dc","Type":"ContainerStarted","Data":"c4362b87c6e9b5388bce020141e751ef24c95d17c419f19b07935a4f9258b2bd"} Jul 16 21:50:03.333356 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.333285 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dmp8k" event={"ID":"079499fc-0421-434a-8a61-d0d075b89be1","Type":"ContainerStarted","Data":"5c96d284a34b7fe6a96909b68cdf805ae4062886a7075e894052c094ac018dd5"} Jul 16 21:50:03.336341 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.336307 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5x7sg" event={"ID":"cd481684-0c85-4cbe-ab71-3e6130ab1131","Type":"ContainerStarted","Data":"0780b281d7135e61816635dcef3e205f86031de814604865030da268ad49b624"} Jul 16 21:50:03.342581 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.342544 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b2wz8" event={"ID":"c29a9f8e-4098-45a1-8c53-84c0fc20f0c1","Type":"ContainerStarted","Data":"6f8f914efab06d9f69cfad55f1507019fd438f6e04e95a399cdb31396584abb6"} Jul 16 21:50:03.354811 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.354773 2558 generic.go:356] "Generic (PLEG): container finished" podID="73ccc2381eb0957c4c8d3c3bbac73808" containerID="d28d0d15eca24935ff9c61d5ca3197dda88cc826dec9ac0bcc2296d0fb250773" exitCode=0 Jul 16 21:50:03.354992 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.354889 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" event={"ID":"73ccc2381eb0957c4c8d3c3bbac73808","Type":"ContainerDied","Data":"d28d0d15eca24935ff9c61d5ca3197dda88cc826dec9ac0bcc2296d0fb250773"} Jul 16 21:50:03.357342 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.357304 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bzncd" event={"ID":"028411f0-adf7-453e-bb59-1b27079457e4","Type":"ContainerStarted","Data":"d5dadbf8802bd66d323d84227d239f04774751f107847c964a814d93f9c897b1"} Jul 16 21:50:03.363103 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.363068 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8stlx" event={"ID":"3347b12c-12e0-4f72-a9eb-9480db6fd042","Type":"ContainerStarted","Data":"82ea27551de6d1dd5eae14972e7d608d28f65ce18169fcf5adee14819cfe700f"} Jul 16 21:50:03.379392 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:03.379337 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-231.ec2.internal" podStartSLOduration=2.379315019 podStartE2EDuration="2.379315019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:03.333316003 +0000 UTC m=+4.593226433" watchObservedRunningTime="2026-07-16 21:50:03.379315019 +0000 UTC m=+4.639225448" Jul 16 21:50:04.256705 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:04.255971 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:04.256705 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:04.256123 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:04.256705 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:04.256500 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:04.256705 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:04.256608 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:04.375486 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:04.375275 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" event={"ID":"73ccc2381eb0957c4c8d3c3bbac73808","Type":"ContainerStarted","Data":"d79b6d53bb837e49ebfc8e902fae70529e8dbf30e92efebd9fd00ee6e1526d2f"} Jul 16 21:50:04.837888 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:04.837847 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:04.838106 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:04.837915 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4kv\" (UniqueName: \"kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv\") pod \"network-check-target-nhjhp\" (UID: \"21d8755c-3957-4432-87d7-312fb00a1d6a\") " pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:04.838106 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:04.837966 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:04.838106 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:04.838088 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:04.838354 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:04.838158 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret podName:9c1ed479-ddc7-4099-b273-ddd8c98534b4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:08.838135487 +0000 UTC m=+10.098045909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret") pod "global-pull-secret-syncer-jkdtg" (UID: "9c1ed479-ddc7-4099-b273-ddd8c98534b4") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:04.838619 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:04.838588 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:04.838716 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:04.838641 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs podName:7e676b3e-e7cf-41d1-ac8c-e2177080ea66 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:08.838624661 +0000 UTC m=+10.098535066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs") pod "network-metrics-daemon-wkshg" (UID: "7e676b3e-e7cf-41d1-ac8c-e2177080ea66") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:04.838789 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:04.838720 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:50:04.838789 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:04.838736 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:50:04.838789 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:04.838750 2558 projected.go:196] Error preparing data for projected volume kube-api-access-tw4kv for pod openshift-network-diagnostics/network-check-target-nhjhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:04.838972 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:04.838791 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv podName:21d8755c-3957-4432-87d7-312fb00a1d6a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:08.838777668 +0000 UTC m=+10.098688086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tw4kv" (UniqueName: "kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv") pod "network-check-target-nhjhp" (UID: "21d8755c-3957-4432-87d7-312fb00a1d6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:05.259324 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:05.258733 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:05.259324 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:05.258870 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:06.256595 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:06.256560 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:06.256780 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:06.256704 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:06.257140 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:06.257118 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:06.257262 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:06.257242 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:07.257084 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:07.257047 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:07.257622 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:07.257198 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:08.256225 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:08.256191 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:08.256406 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:08.256324 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:08.256764 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:08.256734 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:08.256901 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:08.256850 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:08.877093 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:08.876911 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:08.877093 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:08.876977 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:08.877093 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:08.877010 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4kv\" (UniqueName: \"kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv\") pod \"network-check-target-nhjhp\" (UID: \"21d8755c-3957-4432-87d7-312fb00a1d6a\") " pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:08.877597 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:08.877112 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:08.877597 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:08.877129 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:08.877597 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:08.877145 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:50:08.877597 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:08.877162 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:50:08.877597 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:08.877175 2558 projected.go:196] Error preparing data for projected volume kube-api-access-tw4kv for pod openshift-network-diagnostics/network-check-target-nhjhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:08.877597 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:08.877185 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret podName:9c1ed479-ddc7-4099-b273-ddd8c98534b4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:16.877165764 +0000 UTC m=+18.137076177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret") pod "global-pull-secret-syncer-jkdtg" (UID: "9c1ed479-ddc7-4099-b273-ddd8c98534b4") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:08.877597 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:08.877204 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs podName:7e676b3e-e7cf-41d1-ac8c-e2177080ea66 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:16.877195131 +0000 UTC m=+18.137105538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs") pod "network-metrics-daemon-wkshg" (UID: "7e676b3e-e7cf-41d1-ac8c-e2177080ea66") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:08.877597 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:08.877226 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv podName:21d8755c-3957-4432-87d7-312fb00a1d6a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:16.877209406 +0000 UTC m=+18.137119810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tw4kv" (UniqueName: "kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv") pod "network-check-target-nhjhp" (UID: "21d8755c-3957-4432-87d7-312fb00a1d6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:09.260581 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:09.260550 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:09.260744 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:09.260698 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:10.256500 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:10.256460 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:10.257028 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:10.256511 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:10.257028 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:10.256608 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:10.257028 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:10.256893 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:11.260839 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:11.260806 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:11.261325 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:11.260946 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:12.255974 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:12.255934 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:12.256148 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:12.255943 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:12.256148 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:12.256054 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:12.256148 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:12.256120 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:13.259161 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:13.259133 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:13.259580 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:13.259247 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:14.255727 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:14.255644 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:14.255727 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:14.255664 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:14.255945 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:14.255778 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:14.255945 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:14.255927 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:15.259336 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:15.259309 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:15.259731 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:15.259430 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:16.256163 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:16.256130 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:16.256371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:16.256194 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:16.256371 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:16.256326 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:16.256498 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:16.256459 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:16.937723 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:16.937683 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:16.938305 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:16.937753 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:16.938305 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:16.937801 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4kv\" (UniqueName: \"kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv\") pod \"network-check-target-nhjhp\" (UID: \"21d8755c-3957-4432-87d7-312fb00a1d6a\") " pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:16.938305 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:16.937887 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:16.938305 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:16.937922 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:16.938305 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:16.937944 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:50:16.938305 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:16.937965 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:50:16.938305 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:16.937978 2558 projected.go:196] Error preparing data for projected volume kube-api-access-tw4kv for pod openshift-network-diagnostics/network-check-target-nhjhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:16.938305 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:16.937949 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs podName:7e676b3e-e7cf-41d1-ac8c-e2177080ea66 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:32.937930478 +0000 UTC m=+34.197840881 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs") pod "network-metrics-daemon-wkshg" (UID: "7e676b3e-e7cf-41d1-ac8c-e2177080ea66") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:16.938305 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:16.938037 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret podName:9c1ed479-ddc7-4099-b273-ddd8c98534b4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:32.938021865 +0000 UTC m=+34.197932266 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret") pod "global-pull-secret-syncer-jkdtg" (UID: "9c1ed479-ddc7-4099-b273-ddd8c98534b4") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:16.938305 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:16.938062 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv podName:21d8755c-3957-4432-87d7-312fb00a1d6a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:32.938052694 +0000 UTC m=+34.197963097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tw4kv" (UniqueName: "kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv") pod "network-check-target-nhjhp" (UID: "21d8755c-3957-4432-87d7-312fb00a1d6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:17.256505 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:17.256415 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:17.256660 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:17.256551 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:18.255755 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:18.255559 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:18.255755 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:18.255669 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:18.255755 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:18.255721 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:18.256369 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:18.255821 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:19.258804 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:19.258780 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:19.259191 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:19.258893 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:20.255914 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:20.255882 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:20.256091 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:20.255882 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:20.256091 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:20.256007 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:20.256091 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:20.256066 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:21.259167 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:21.259141 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:21.259583 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:21.259248 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:22.256106 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.255700 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:22.256191 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.255754 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:22.256278 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:22.256229 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:22.256533 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:22.256515 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:22.421303 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.421267 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bzncd" event={"ID":"028411f0-adf7-453e-bb59-1b27079457e4","Type":"ContainerStarted","Data":"e43a2fbeb662fc5be9eda5dd99458fddc08fdfccc94601455b3c56d51139af27"} Jul 16 21:50:22.423040 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.423016 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8stlx" event={"ID":"3347b12c-12e0-4f72-a9eb-9480db6fd042","Type":"ContainerStarted","Data":"d8d0a22e8b8ef971700ac107437077f6216ceaab099fb18e58630ed580687003"} Jul 16 21:50:22.424482 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.424457 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" event={"ID":"b8d78eee-2888-4499-a24b-c2bc1123a5a2","Type":"ContainerStarted","Data":"6d69b26d8ffd392a6037d2b4e176ea5f99bc591b084d42a0149a1612b865f216"} Jul 16 21:50:22.427005 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.426981 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" event={"ID":"a7fe9137-5027-40c2-aebf-9c0a64d1fe37","Type":"ContainerStarted","Data":"6687abffc28792dded035a759747f430fb4d347b07cd9d290780556ab6845c3b"} Jul 16 21:50:22.427109 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.427040 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" event={"ID":"a7fe9137-5027-40c2-aebf-9c0a64d1fe37","Type":"ContainerStarted","Data":"cb98f989a28c71752e15a8e2e54355e905da29d2340cc507e4a94d1ebee39fc9"} Jul 16 21:50:22.427109 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.427071 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" event={"ID":"a7fe9137-5027-40c2-aebf-9c0a64d1fe37","Type":"ContainerStarted","Data":"af040fa02e9a533a1043a7dc11ee1a591ea6328639961f67a7028c3f2517eed4"} Jul 16 21:50:22.428248 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.428229 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" event={"ID":"e866eb17-d39c-4e40-87b2-dd1ab0d3133d","Type":"ContainerStarted","Data":"b1a864937d73823e9bbc3f505e7358cb761a5203c51b93af795cf8c3455be225"} Jul 16 21:50:22.429561 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.429542 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dmp8k" event={"ID":"079499fc-0421-434a-8a61-d0d075b89be1","Type":"ContainerStarted","Data":"f4b41a07095425c4f758488964850df55050c8543bd7ce21ea60122ebfaa4cd8"} Jul 16 21:50:22.430776 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.430757 2558 generic.go:356] "Generic (PLEG): container finished" podID="cd481684-0c85-4cbe-ab71-3e6130ab1131" containerID="48b21677e9660e95898b246006a06cd891164cc58f7858b34aa5ce2e64891518" exitCode=0 Jul 16 21:50:22.430888 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.430835 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5x7sg" event={"ID":"cd481684-0c85-4cbe-ab71-3e6130ab1131","Type":"ContainerDied","Data":"48b21677e9660e95898b246006a06cd891164cc58f7858b34aa5ce2e64891518"} Jul 16 21:50:22.432346 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.432325 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b2wz8" event={"ID":"c29a9f8e-4098-45a1-8c53-84c0fc20f0c1","Type":"ContainerStarted","Data":"9b03993487f86cfd685f7b4aba10cab4e95261d0b24f724811463d3de5dd7ac0"} Jul 16 21:50:22.448384 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.448343 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-231.ec2.internal" podStartSLOduration=21.448332061 podStartE2EDuration="21.448332061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:04.393066158 +0000 UTC m=+5.652976600" watchObservedRunningTime="2026-07-16 21:50:22.448332061 +0000 UTC m=+23.708242484" Jul 16 21:50:22.448481 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.448464 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bzncd" podStartSLOduration=2.741427523 podStartE2EDuration="22.448461044s" totalImagesPullingTime="19.707033521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:22.448203722 +0000 UTC m=+23.708114146" watchObservedRunningTime="2026-07-16 21:50:22.448461044 +0000 UTC m=+23.708371468" Jul 16 21:50:22.487641 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.487598 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dmp8k" podStartSLOduration=2.842290736 podStartE2EDuration="22.487579667s" totalImagesPullingTime="19.645288931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:22.487316702 +0000 UTC m=+23.747227128" watchObservedRunningTime="2026-07-16 21:50:22.487579667 +0000 UTC m=+23.747490159" Jul 16 21:50:22.526588 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.526550 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sx6fl" podStartSLOduration=2.847120181 podStartE2EDuration="22.526536115s" totalImagesPullingTime="19.679415934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:22.509494169 +0000 UTC m=+23.769404593" watchObservedRunningTime="2026-07-16 21:50:22.526536115 +0000 UTC m=+23.786446538" Jul 16 21:50:22.526682 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.526642 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b2wz8" podStartSLOduration=2.866932145 podStartE2EDuration="22.526637691s" totalImagesPullingTime="19.659705546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:22.526209797 +0000 UTC m=+23.786120240" watchObservedRunningTime="2026-07-16 21:50:22.526637691 +0000 UTC m=+23.786548114" Jul 16 21:50:22.547122 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:22.547086 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8stlx" podStartSLOduration=2.9042145230000003 podStartE2EDuration="22.547071539s" totalImagesPullingTime="19.642857016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:22.547036344 +0000 UTC m=+23.806946768" watchObservedRunningTime="2026-07-16 21:50:22.547071539 +0000 UTC m=+23.806981962" Jul 16 21:50:23.231141 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:23.231056 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmp8k_079499fc-0421-434a-8a61-d0d075b89be1/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:23.256104 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:23.256067 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:23.256272 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:23.256197 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:23.340831 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:23.340800 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmp8k_079499fc-0421-434a-8a61-d0d075b89be1/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:23.441426 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:23.441393 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" event={"ID":"a7fe9137-5027-40c2-aebf-9c0a64d1fe37","Type":"ContainerStarted","Data":"8877bbff0ab0a01edbf562e34d185b8464553174869dc65fa798c1911ae168e3"} Jul 16 21:50:23.441874 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:23.441492 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" event={"ID":"a7fe9137-5027-40c2-aebf-9c0a64d1fe37","Type":"ContainerStarted","Data":"188f51dd12bd8e82083a07641e9bb9a66525518b30fbb4aedd1d5f76ed5795a9"} Jul 16 21:50:23.441874 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:23.441543 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" event={"ID":"a7fe9137-5027-40c2-aebf-9c0a64d1fe37","Type":"ContainerStarted","Data":"0a5aae480cd31583ad7c443b779579a03a398969b2fe378dd9caf3d23a3d58ef"} Jul 16 21:50:23.443910 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:23.443883 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nhs4l" event={"ID":"ca08e685-21d1-4a2a-8c21-5579a3ccb8dc","Type":"ContainerStarted","Data":"2141aa7cc3afa2d31d29368058e3c32674636b483ae0bd813e397e3ec7982e99"} Jul 16 21:50:23.464454 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:23.464399 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nhs4l" podStartSLOduration=3.808956122 podStartE2EDuration="23.464380686s" totalImagesPullingTime="19.655424564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:23.463806857 +0000 UTC m=+24.723717284" watchObservedRunningTime="2026-07-16 21:50:23.464380686 +0000 UTC m=+24.724291112" Jul 16 21:50:23.625518 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:23.625480 2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 21:50:23.628380 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:23.628356 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmp8k_079499fc-0421-434a-8a61-d0d075b89be1/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:23.723582 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:23.723512 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmp8k_079499fc-0421-434a-8a61-d0d075b89be1/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:24.154625 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:24.154593 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmp8k_079499fc-0421-434a-8a61-d0d075b89be1/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:24.172539 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:24.172423 2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:50:23.625505736Z","UUID":"1857e42d-0f12-4b03-bf83-6ed0eb65a8e0","Handler":null,"Name":"","Endpoint":""} Jul 16 21:50:24.175053 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:24.175028 2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 21:50:24.175174 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:24.175079 2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 21:50:24.256115 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:24.256085 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:24.256290 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:24.256123 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:24.256290 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:24.256191 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:24.256432 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:24.256324 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:24.257719 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:24.257684 2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dmp8k_079499fc-0421-434a-8a61-d0d075b89be1/konnectivity-agent/0.log" limit=1024 Jul 16 21:50:24.448358 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:24.448319 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" event={"ID":"b8d78eee-2888-4499-a24b-c2bc1123a5a2","Type":"ContainerStarted","Data":"1ddb254c6ea8cebe53f516fc520beb6cece83939fbbbd2e5d04e40cef67ab6bc"} Jul 16 21:50:25.256114 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:25.256080 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:25.256319 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:25.256210 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:25.452707 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:25.452673 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" event={"ID":"b8d78eee-2888-4499-a24b-c2bc1123a5a2","Type":"ContainerStarted","Data":"59afa2b7e9a3a55f3dc7e30f0ba881eeadcdee017a7b56747148421c6589bef4"} Jul 16 21:50:25.456384 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:25.456352 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" event={"ID":"a7fe9137-5027-40c2-aebf-9c0a64d1fe37","Type":"ContainerStarted","Data":"f4942218258f7cd10254a48ef729e42996daba1d318e950cea8403ffd66bce09"} Jul 16 21:50:25.490215 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:25.490168 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d9glc" podStartSLOduration=3.505533656 podStartE2EDuration="25.490148161s" totalImagesPullingTime="21.984614505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:25.489976319 +0000 UTC m=+26.749886764" watchObservedRunningTime="2026-07-16 21:50:25.490148161 +0000 UTC m=+26.750058587" Jul 16 21:50:26.256271 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:26.256083 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:26.256444 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:26.256105 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:26.256444 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:26.256372 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:26.256547 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:26.256444 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:27.204093 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:27.203900 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:27.204814 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:27.204525 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:27.256388 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:27.256362 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:27.256542 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:27.256499 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:27.464425 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:27.464336 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" event={"ID":"a7fe9137-5027-40c2-aebf-9c0a64d1fe37","Type":"ContainerStarted","Data":"4ac52f859f1c245e558602f8cf7dd1f30dd3deba1c73fb93084132678526f5fc"} Jul 16 21:50:27.464602 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:27.464574 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:27.466122 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:27.466102 2558 generic.go:356] "Generic (PLEG): container finished" podID="cd481684-0c85-4cbe-ab71-3e6130ab1131" containerID="7ea9f9f585eb07a79523190696f09dc1f8af2d011d8c109b39332aeb8c3cb298" exitCode=0 Jul 16 21:50:27.466223 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:27.466188 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5x7sg" event={"ID":"cd481684-0c85-4cbe-ab71-3e6130ab1131","Type":"ContainerDied","Data":"7ea9f9f585eb07a79523190696f09dc1f8af2d011d8c109b39332aeb8c3cb298"} Jul 16 21:50:27.466365 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:27.466345 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:27.467176 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:27.467111 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dmp8k" Jul 16 21:50:27.481755 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:27.481729 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:27.531395 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:27.531351 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" podStartSLOduration=7.813484501 podStartE2EDuration="27.531334582s" totalImagesPullingTime="19.717850081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:27.502998569 +0000 UTC m=+28.762909013" watchObservedRunningTime="2026-07-16 21:50:27.531334582 +0000 UTC m=+28.791245022" Jul 16 21:50:28.257303 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:28.255750 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:28.257303 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:28.255895 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:28.257303 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:28.256256 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:28.257303 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:28.256346 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:28.469558 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:28.468461 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:28.469558 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:28.468553 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:28.484919 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:28.484888 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:50:28.933875 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:28.933821 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nhjhp"] Jul 16 21:50:28.934067 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:28.933973 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:28.934120 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:28.934082 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:28.934473 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:28.934456 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jkdtg"] Jul 16 21:50:28.934534 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:28.934525 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:28.934602 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:28.934587 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:28.936122 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:28.936100 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wkshg"] Jul 16 21:50:28.936221 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:28.936205 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:28.936306 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:28.936290 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:29.472755 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:29.472722 2558 generic.go:356] "Generic (PLEG): container finished" podID="cd481684-0c85-4cbe-ab71-3e6130ab1131" containerID="1c1d456a73eeb670e2603483b05b10b840509b79aaacf1e3da9a563a4a5c5b45" exitCode=0 Jul 16 21:50:29.473170 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:29.472814 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5x7sg" event={"ID":"cd481684-0c85-4cbe-ab71-3e6130ab1131","Type":"ContainerDied","Data":"1c1d456a73eeb670e2603483b05b10b840509b79aaacf1e3da9a563a4a5c5b45"} Jul 16 21:50:30.256708 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:30.256679 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:30.256913 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:30.256810 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:30.256913 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:30.256885 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:30.257038 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:30.257019 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:31.256171 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:31.255925 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:31.256545 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:31.256206 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:31.480263 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:31.480224 2558 generic.go:356] "Generic (PLEG): container finished" podID="cd481684-0c85-4cbe-ab71-3e6130ab1131" containerID="68d82f0e9904da725dcf61a4eeda80cedb2a410f15717c9d916e3908bc444de9" exitCode=0 Jul 16 21:50:31.480442 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:31.480294 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5x7sg" event={"ID":"cd481684-0c85-4cbe-ab71-3e6130ab1131","Type":"ContainerDied","Data":"68d82f0e9904da725dcf61a4eeda80cedb2a410f15717c9d916e3908bc444de9"} Jul 16 21:50:32.256101 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:32.256066 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:32.256280 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:32.256121 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:32.256280 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:32.256199 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:32.256728 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:32.256336 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:32.955706 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:32.955672 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:32.955959 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:32.955732 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:32.955959 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:32.955833 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:32.955959 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:32.955837 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:32.955959 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:32.955924 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret podName:9c1ed479-ddc7-4099-b273-ddd8c98534b4 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:04.955902346 +0000 UTC m=+66.215812751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret") pod "global-pull-secret-syncer-jkdtg" (UID: "9c1ed479-ddc7-4099-b273-ddd8c98534b4") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:32.955959 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:32.955944 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs podName:7e676b3e-e7cf-41d1-ac8c-e2177080ea66 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:04.955933311 +0000 UTC m=+66.215843717 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs") pod "network-metrics-daemon-wkshg" (UID: "7e676b3e-e7cf-41d1-ac8c-e2177080ea66") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:32.956223 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:32.955970 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4kv\" (UniqueName: \"kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv\") pod \"network-check-target-nhjhp\" (UID: \"21d8755c-3957-4432-87d7-312fb00a1d6a\") " pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:32.956223 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:32.956087 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:50:32.956223 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:32.956098 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:50:32.956223 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:32.956107 2558 projected.go:196] Error preparing data for projected volume kube-api-access-tw4kv for pod openshift-network-diagnostics/network-check-target-nhjhp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:32.956223 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:32.956134 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv podName:21d8755c-3957-4432-87d7-312fb00a1d6a nodeName:}" failed. No retries permitted until 2026-07-16 21:51:04.956126715 +0000 UTC m=+66.216037116 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tw4kv" (UniqueName: "kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv") pod "network-check-target-nhjhp" (UID: "21d8755c-3957-4432-87d7-312fb00a1d6a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:33.256018 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:33.255933 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:33.256162 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:33.256076 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wkshg" podUID="7e676b3e-e7cf-41d1-ac8c-e2177080ea66" Jul 16 21:50:34.256345 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.256309 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:34.256345 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.256339 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:34.257001 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:34.256453 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jkdtg" podUID="9c1ed479-ddc7-4099-b273-ddd8c98534b4" Jul 16 21:50:34.257001 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:34.256565 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhjhp" podUID="21d8755c-3957-4432-87d7-312fb00a1d6a" Jul 16 21:50:34.587084 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.587057 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-231.ec2.internal" event="NodeReady" Jul 16 21:50:34.587236 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.587198 2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 21:50:34.641019 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.640973 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xgq9j"] Jul 16 21:50:34.643510 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.643486 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.648744 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.648722 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 21:50:34.648744 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.648740 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 21:50:34.648971 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.648804 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6n6rt\"" Jul 16 21:50:34.653530 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.653514 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mz7fk"] Jul 16 21:50:34.655304 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.655288 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mz7fk" Jul 16 21:50:34.656312 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.656295 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xgq9j"] Jul 16 21:50:34.658062 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.658037 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 21:50:34.658157 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.658037 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pz5gf\"" Jul 16 21:50:34.658157 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.658038 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 21:50:34.658157 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.658072 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 21:50:34.670166 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.669987 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mz7fk"] Jul 16 21:50:34.771367 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.771285 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0c065d-3277-4995-beca-874814b115fa-tmp-dir\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.771367 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.771331 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scd4w\" (UniqueName: \"kubernetes.io/projected/cad724ee-e0b4-4519-9f88-52e577024ced-kube-api-access-scd4w\") pod \"ingress-canary-mz7fk\" (UID: \"cad724ee-e0b4-4519-9f88-52e577024ced\") " pod="openshift-ingress-canary/ingress-canary-mz7fk" Jul 16 21:50:34.771574 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.771398 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0c065d-3277-4995-beca-874814b115fa-config-volume\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.771574 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.771426 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.771574 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.771482 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45j8p\" (UniqueName: \"kubernetes.io/projected/3c0c065d-3277-4995-beca-874814b115fa-kube-api-access-45j8p\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.771574 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.771548 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cad724ee-e0b4-4519-9f88-52e577024ced-cert\") pod \"ingress-canary-mz7fk\" (UID: \"cad724ee-e0b4-4519-9f88-52e577024ced\") " pod="openshift-ingress-canary/ingress-canary-mz7fk" Jul 16 21:50:34.872709 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.872666 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0c065d-3277-4995-beca-874814b115fa-config-volume\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.872923 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.872720 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.872923 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.872750 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45j8p\" (UniqueName: \"kubernetes.io/projected/3c0c065d-3277-4995-beca-874814b115fa-kube-api-access-45j8p\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.872923 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:34.872872 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:34.873085 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.872965 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cad724ee-e0b4-4519-9f88-52e577024ced-cert\") pod \"ingress-canary-mz7fk\" (UID: \"cad724ee-e0b4-4519-9f88-52e577024ced\") " pod="openshift-ingress-canary/ingress-canary-mz7fk" Jul 16 21:50:34.873085 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.872996 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0c065d-3277-4995-beca-874814b115fa-tmp-dir\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.873085 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:34.873043 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls podName:3c0c065d-3277-4995-beca-874814b115fa nodeName:}" failed. No retries permitted until 2026-07-16 21:50:35.373023603 +0000 UTC m=+36.632934007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls") pod "dns-default-xgq9j" (UID: "3c0c065d-3277-4995-beca-874814b115fa") : secret "dns-default-metrics-tls" not found Jul 16 21:50:34.873085 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.873060 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scd4w\" (UniqueName: \"kubernetes.io/projected/cad724ee-e0b4-4519-9f88-52e577024ced-kube-api-access-scd4w\") pod \"ingress-canary-mz7fk\" (UID: \"cad724ee-e0b4-4519-9f88-52e577024ced\") " pod="openshift-ingress-canary/ingress-canary-mz7fk" Jul 16 21:50:34.873337 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.873316 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c0c065d-3277-4995-beca-874814b115fa-tmp-dir\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.873390 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.873370 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0c065d-3277-4995-beca-874814b115fa-config-volume\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.877884 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.877834 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cad724ee-e0b4-4519-9f88-52e577024ced-cert\") pod \"ingress-canary-mz7fk\" (UID: \"cad724ee-e0b4-4519-9f88-52e577024ced\") " pod="openshift-ingress-canary/ingress-canary-mz7fk" Jul 16 21:50:34.881801 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.881775 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scd4w\" (UniqueName: \"kubernetes.io/projected/cad724ee-e0b4-4519-9f88-52e577024ced-kube-api-access-scd4w\") pod \"ingress-canary-mz7fk\" (UID: \"cad724ee-e0b4-4519-9f88-52e577024ced\") " pod="openshift-ingress-canary/ingress-canary-mz7fk" Jul 16 21:50:34.881994 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.881974 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45j8p\" (UniqueName: \"kubernetes.io/projected/3c0c065d-3277-4995-beca-874814b115fa-kube-api-access-45j8p\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:34.965200 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:34.965160 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mz7fk" Jul 16 21:50:35.256710 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:35.256676 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:50:35.263132 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:35.263106 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xshkd\"" Jul 16 21:50:35.263558 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:35.263533 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 21:50:35.377263 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:35.377223 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:35.377417 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:35.377340 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:35.377417 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:35.377414 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls podName:3c0c065d-3277-4995-beca-874814b115fa nodeName:}" failed. No retries permitted until 2026-07-16 21:50:36.377399979 +0000 UTC m=+37.637310382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls") pod "dns-default-xgq9j" (UID: "3c0c065d-3277-4995-beca-874814b115fa") : secret "dns-default-metrics-tls" not found Jul 16 21:50:35.617120 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:35.616918 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mz7fk"] Jul 16 21:50:35.621660 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:50:35.621630 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad724ee_e0b4_4519_9f88_52e577024ced.slice/crio-1b61846c43501df1fa5f1c5cce9d1d19df6f0dba0392b9f1e7c7ff6aca2b91dd WatchSource:0}: Error finding container 1b61846c43501df1fa5f1c5cce9d1d19df6f0dba0392b9f1e7c7ff6aca2b91dd: Status 404 returned error can't find the container with id 1b61846c43501df1fa5f1c5cce9d1d19df6f0dba0392b9f1e7c7ff6aca2b91dd Jul 16 21:50:36.256457 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:36.256423 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:50:36.256664 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:36.256423 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:50:36.259321 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:36.259301 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 21:50:36.259664 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:36.259309 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 21:50:36.260169 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:36.260151 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 21:50:36.260272 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:36.260154 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqtjl\"" Jul 16 21:50:36.387811 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:36.387774 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:36.388001 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:36.387912 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:36.388001 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:36.387972 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls podName:3c0c065d-3277-4995-beca-874814b115fa nodeName:}" failed. No retries permitted until 2026-07-16 21:50:38.387953661 +0000 UTC m=+39.647864063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls") pod "dns-default-xgq9j" (UID: "3c0c065d-3277-4995-beca-874814b115fa") : secret "dns-default-metrics-tls" not found Jul 16 21:50:36.493267 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:36.493229 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mz7fk" event={"ID":"cad724ee-e0b4-4519-9f88-52e577024ced","Type":"ContainerStarted","Data":"1b61846c43501df1fa5f1c5cce9d1d19df6f0dba0392b9f1e7c7ff6aca2b91dd"} Jul 16 21:50:36.496346 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:36.496309 2558 generic.go:356] "Generic (PLEG): container finished" podID="cd481684-0c85-4cbe-ab71-3e6130ab1131" containerID="10c501dbfa43cdd1155127e1dbe61032937c7977648e9a5749f41b4353f4be6f" exitCode=0 Jul 16 21:50:36.496469 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:36.496363 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5x7sg" event={"ID":"cd481684-0c85-4cbe-ab71-3e6130ab1131","Type":"ContainerDied","Data":"10c501dbfa43cdd1155127e1dbe61032937c7977648e9a5749f41b4353f4be6f"} Jul 16 21:50:37.502844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:37.502812 2558 generic.go:356] "Generic (PLEG): container finished" podID="cd481684-0c85-4cbe-ab71-3e6130ab1131" containerID="0a168b676be4a516ecba97baa1d9f5550907c648ee96ba19f9f0aaf1e5ce6411" exitCode=0 Jul 16 21:50:37.503280 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:37.502891 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5x7sg" event={"ID":"cd481684-0c85-4cbe-ab71-3e6130ab1131","Type":"ContainerDied","Data":"0a168b676be4a516ecba97baa1d9f5550907c648ee96ba19f9f0aaf1e5ce6411"} Jul 16 21:50:38.403751 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:38.403577 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:38.403751 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:38.403736 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:38.403994 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:38.403803 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls podName:3c0c065d-3277-4995-beca-874814b115fa nodeName:}" failed. No retries permitted until 2026-07-16 21:50:42.403787933 +0000 UTC m=+43.663698335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls") pod "dns-default-xgq9j" (UID: "3c0c065d-3277-4995-beca-874814b115fa") : secret "dns-default-metrics-tls" not found Jul 16 21:50:38.506385 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:38.506351 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mz7fk" event={"ID":"cad724ee-e0b4-4519-9f88-52e577024ced","Type":"ContainerStarted","Data":"bb2c4135c573aaf9ad949d3b95548594db402cc97a41e2b50dea20c8cda66fd2"} Jul 16 21:50:38.509313 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:38.509286 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5x7sg" event={"ID":"cd481684-0c85-4cbe-ab71-3e6130ab1131","Type":"ContainerStarted","Data":"90f9530773ec41da263b955ff10160539d1d28a3cee03bbf1bfa43e2d4900a18"} Jul 16 21:50:38.524716 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:38.524673 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mz7fk" podStartSLOduration=2.469683454 podStartE2EDuration="4.524658144s" totalImagesPullingTime="2.05497469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:38.524023073 +0000 UTC m=+39.783933510" watchObservedRunningTime="2026-07-16 21:50:38.524658144 +0000 UTC m=+39.784568568" Jul 16 21:50:38.551464 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:38.551428 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5x7sg" podStartSLOduration=8.41210747 podStartE2EDuration="38.551412813s" totalImagesPullingTime="30.139305343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:38.550479535 +0000 UTC m=+39.810389960" watchObservedRunningTime="2026-07-16 21:50:38.551412813 +0000 UTC m=+39.811323237" Jul 16 21:50:42.432981 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:42.432939 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:42.433634 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:42.433109 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:42.433634 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:42.433186 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls podName:3c0c065d-3277-4995-beca-874814b115fa nodeName:}" failed. No retries permitted until 2026-07-16 21:50:50.433165508 +0000 UTC m=+51.693075928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls") pod "dns-default-xgq9j" (UID: "3c0c065d-3277-4995-beca-874814b115fa") : secret "dns-default-metrics-tls" not found Jul 16 21:50:50.490191 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:50.490149 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:50:50.490681 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:50.490295 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:50.490681 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:50.490353 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls podName:3c0c065d-3277-4995-beca-874814b115fa nodeName:}" failed. No retries permitted until 2026-07-16 21:51:06.490338596 +0000 UTC m=+67.750248998 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls") pod "dns-default-xgq9j" (UID: "3c0c065d-3277-4995-beca-874814b115fa") : secret "dns-default-metrics-tls" not found Jul 16 21:50:58.017239 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.017207 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689"] Jul 16 21:50:58.019391 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.019372 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" Jul 16 21:50:58.021614 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.021592 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5rcvp\"" Jul 16 21:50:58.021723 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.021601 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 16 21:50:58.022565 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.022531 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 16 21:50:58.022672 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.022566 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 16 21:50:58.022672 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.022601 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:50:58.028342 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.028321 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689"] Jul 16 21:50:58.127980 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.127951 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4"] Jul 16 21:50:58.129927 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.129912 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t"] Jul 16 21:50:58.130062 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.130046 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" Jul 16 21:50:58.132100 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.132080 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:50:58.132874 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.132844 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 16 21:50:58.133086 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.133071 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:50:58.133325 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.133311 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 16 21:50:58.133510 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.133496 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 16 21:50:58.134125 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.134109 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sb7hj\"" Jul 16 21:50:58.135216 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.135201 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tgxzg\"" Jul 16 21:50:58.135638 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.135622 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 16 21:50:58.136825 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.136664 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:50:58.138195 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.138176 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 16 21:50:58.138384 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.138362 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5j5\" (UniqueName: \"kubernetes.io/projected/37690e6c-a753-4329-8d1b-f327f3cf2c25-kube-api-access-px5j5\") pod \"service-ca-operator-5bcbc94c78-7p689\" (UID: \"37690e6c-a753-4329-8d1b-f327f3cf2c25\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" Jul 16 21:50:58.138487 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.138471 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37690e6c-a753-4329-8d1b-f327f3cf2c25-config\") pod \"service-ca-operator-5bcbc94c78-7p689\" (UID: \"37690e6c-a753-4329-8d1b-f327f3cf2c25\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" Jul 16 21:50:58.138560 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.138542 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37690e6c-a753-4329-8d1b-f327f3cf2c25-serving-cert\") pod \"service-ca-operator-5bcbc94c78-7p689\" (UID: \"37690e6c-a753-4329-8d1b-f327f3cf2c25\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" Jul 16 21:50:58.148174 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.148150 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4"] Jul 16 21:50:58.148699 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.148679 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t"] Jul 16 21:50:58.239414 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.239381 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jw78t\" (UID: \"ee6ee131-6cea-42c7-80c2-02bbf11bcca1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:50:58.239414 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.239413 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzlvx\" (UniqueName: \"kubernetes.io/projected/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-kube-api-access-tzlvx\") pod \"cluster-samples-operator-79b7c55f87-jw78t\" (UID: \"ee6ee131-6cea-42c7-80c2-02bbf11bcca1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:50:58.239659 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.239465 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37690e6c-a753-4329-8d1b-f327f3cf2c25-config\") pod \"service-ca-operator-5bcbc94c78-7p689\" (UID: \"37690e6c-a753-4329-8d1b-f327f3cf2c25\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" Jul 16 21:50:58.239659 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.239518 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37690e6c-a753-4329-8d1b-f327f3cf2c25-serving-cert\") pod \"service-ca-operator-5bcbc94c78-7p689\" (UID: \"37690e6c-a753-4329-8d1b-f327f3cf2c25\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" Jul 16 21:50:58.239659 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.239551 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cd0ba43-b852-4efc-bfb9-0303473bb89c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-s9jq4\" (UID: \"5cd0ba43-b852-4efc-bfb9-0303473bb89c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" Jul 16 21:50:58.239659 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.239593 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5j5\" (UniqueName: \"kubernetes.io/projected/37690e6c-a753-4329-8d1b-f327f3cf2c25-kube-api-access-px5j5\") pod \"service-ca-operator-5bcbc94c78-7p689\" (UID: \"37690e6c-a753-4329-8d1b-f327f3cf2c25\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" Jul 16 21:50:58.239659 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.239623 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhsc\" (UniqueName: \"kubernetes.io/projected/5cd0ba43-b852-4efc-bfb9-0303473bb89c-kube-api-access-2dhsc\") pod \"kube-storage-version-migrator-operator-7658d9b749-s9jq4\" (UID: \"5cd0ba43-b852-4efc-bfb9-0303473bb89c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" Jul 16 21:50:58.239833 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.239675 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5cd0ba43-b852-4efc-bfb9-0303473bb89c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-s9jq4\" (UID: \"5cd0ba43-b852-4efc-bfb9-0303473bb89c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" Jul 16 21:50:58.240026 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.240000 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/37690e6c-a753-4329-8d1b-f327f3cf2c25-config\") pod \"service-ca-operator-5bcbc94c78-7p689\" (UID: \"37690e6c-a753-4329-8d1b-f327f3cf2c25\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" Jul 16 21:50:58.243675 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.243658 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/37690e6c-a753-4329-8d1b-f327f3cf2c25-serving-cert\") pod \"service-ca-operator-5bcbc94c78-7p689\" (UID: \"37690e6c-a753-4329-8d1b-f327f3cf2c25\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" Jul 16 21:50:58.247567 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.247547 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c99d8d4d4-4trjg"] Jul 16 21:50:58.252539 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.252525 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.254872 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.254834 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5j5\" (UniqueName: \"kubernetes.io/projected/37690e6c-a753-4329-8d1b-f327f3cf2c25-kube-api-access-px5j5\") pod \"service-ca-operator-5bcbc94c78-7p689\" (UID: \"37690e6c-a753-4329-8d1b-f327f3cf2c25\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" Jul 16 21:50:58.255140 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.255123 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 16 21:50:58.255540 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.255525 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gcvjv\"" Jul 16 21:50:58.255754 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.255740 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 21:50:58.255826 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.255811 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 16 21:50:58.255895 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.255884 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 21:50:58.256156 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.256142 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 16 21:50:58.262407 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.262390 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 16 21:50:58.272650 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.272603 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c99d8d4d4-4trjg"] Jul 16 21:50:58.327971 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.327948 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" Jul 16 21:50:58.341021 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.340997 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhsc\" (UniqueName: \"kubernetes.io/projected/5cd0ba43-b852-4efc-bfb9-0303473bb89c-kube-api-access-2dhsc\") pod \"kube-storage-version-migrator-operator-7658d9b749-s9jq4\" (UID: \"5cd0ba43-b852-4efc-bfb9-0303473bb89c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" Jul 16 21:50:58.341141 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.341065 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5cd0ba43-b852-4efc-bfb9-0303473bb89c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-s9jq4\" (UID: \"5cd0ba43-b852-4efc-bfb9-0303473bb89c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" Jul 16 21:50:58.341141 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.341109 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jw78t\" (UID: \"ee6ee131-6cea-42c7-80c2-02bbf11bcca1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:50:58.341141 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.341135 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzlvx\" (UniqueName: \"kubernetes.io/projected/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-kube-api-access-tzlvx\") pod \"cluster-samples-operator-79b7c55f87-jw78t\" (UID: \"ee6ee131-6cea-42c7-80c2-02bbf11bcca1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:50:58.341296 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.341192 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cd0ba43-b852-4efc-bfb9-0303473bb89c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-s9jq4\" (UID: \"5cd0ba43-b852-4efc-bfb9-0303473bb89c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" Jul 16 21:50:58.341441 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:58.341408 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 21:50:58.341516 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:58.341478 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls podName:ee6ee131-6cea-42c7-80c2-02bbf11bcca1 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:58.841457903 +0000 UTC m=+60.101368328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jw78t" (UID: "ee6ee131-6cea-42c7-80c2-02bbf11bcca1") : secret "samples-operator-tls" not found Jul 16 21:50:58.341691 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.341669 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5cd0ba43-b852-4efc-bfb9-0303473bb89c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-s9jq4\" (UID: \"5cd0ba43-b852-4efc-bfb9-0303473bb89c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" Jul 16 21:50:58.343955 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.343936 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5cd0ba43-b852-4efc-bfb9-0303473bb89c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-s9jq4\" (UID: \"5cd0ba43-b852-4efc-bfb9-0303473bb89c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" Jul 16 21:50:58.356656 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.356629 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzlvx\" (UniqueName: \"kubernetes.io/projected/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-kube-api-access-tzlvx\") pod \"cluster-samples-operator-79b7c55f87-jw78t\" (UID: \"ee6ee131-6cea-42c7-80c2-02bbf11bcca1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:50:58.357647 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.357631 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhsc\" (UniqueName: \"kubernetes.io/projected/5cd0ba43-b852-4efc-bfb9-0303473bb89c-kube-api-access-2dhsc\") pod \"kube-storage-version-migrator-operator-7658d9b749-s9jq4\" (UID: \"5cd0ba43-b852-4efc-bfb9-0303473bb89c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" Jul 16 21:50:58.442351 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.442321 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-stats-auth\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.442482 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.442383 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.442482 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.442402 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.442482 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.442430 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg4bx\" (UniqueName: \"kubernetes.io/projected/5db1862a-2a05-4cf2-bc16-05f4d1188f88-kube-api-access-mg4bx\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.442482 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.442451 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-default-certificate\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.443122 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.443104 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" Jul 16 21:50:58.464518 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.464495 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689"] Jul 16 21:50:58.467635 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:50:58.467612 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37690e6c_a753_4329_8d1b_f327f3cf2c25.slice/crio-d05748099fa99c9d4c6abc2d2c81cca64079a13c7a37ede2382818e418ac06f6 WatchSource:0}: Error finding container d05748099fa99c9d4c6abc2d2c81cca64079a13c7a37ede2382818e418ac06f6: Status 404 returned error can't find the container with id d05748099fa99c9d4c6abc2d2c81cca64079a13c7a37ede2382818e418ac06f6 Jul 16 21:50:58.543131 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.543101 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg4bx\" (UniqueName: \"kubernetes.io/projected/5db1862a-2a05-4cf2-bc16-05f4d1188f88-kube-api-access-mg4bx\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.543233 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.543164 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-default-certificate\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.543295 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.543232 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-stats-auth\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.543350 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.543317 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.543402 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.543348 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.543565 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:58.543512 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle podName:5db1862a-2a05-4cf2-bc16-05f4d1188f88 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:59.043490904 +0000 UTC m=+60.303401329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle") pod "router-default-c99d8d4d4-4trjg" (UID: "5db1862a-2a05-4cf2-bc16-05f4d1188f88") : configmap references non-existent config key: service-ca.crt Jul 16 21:50:58.543825 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:58.543805 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 21:50:58.543917 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:58.543884 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs podName:5db1862a-2a05-4cf2-bc16-05f4d1188f88 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:59.043848164 +0000 UTC m=+60.303758570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs") pod "router-default-c99d8d4d4-4trjg" (UID: "5db1862a-2a05-4cf2-bc16-05f4d1188f88") : secret "router-metrics-certs-default" not found Jul 16 21:50:58.546261 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.546240 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-stats-auth\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.546353 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.546248 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-default-certificate\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.557652 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.557585 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" event={"ID":"37690e6c-a753-4329-8d1b-f327f3cf2c25","Type":"ContainerStarted","Data":"d05748099fa99c9d4c6abc2d2c81cca64079a13c7a37ede2382818e418ac06f6"} Jul 16 21:50:58.564138 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.564115 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg4bx\" (UniqueName: \"kubernetes.io/projected/5db1862a-2a05-4cf2-bc16-05f4d1188f88-kube-api-access-mg4bx\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:58.615438 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.615412 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4"] Jul 16 21:50:58.618586 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:50:58.618564 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd0ba43_b852_4efc_bfb9_0303473bb89c.slice/crio-21ae4802c1cf49482210d01c2e3763646fdfaeb84381e24289eb60b1d6128548 WatchSource:0}: Error finding container 21ae4802c1cf49482210d01c2e3763646fdfaeb84381e24289eb60b1d6128548: Status 404 returned error can't find the container with id 21ae4802c1cf49482210d01c2e3763646fdfaeb84381e24289eb60b1d6128548 Jul 16 21:50:58.845732 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:58.845648 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jw78t\" (UID: \"ee6ee131-6cea-42c7-80c2-02bbf11bcca1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:50:58.845916 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:58.845784 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 21:50:58.845916 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:58.845873 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls podName:ee6ee131-6cea-42c7-80c2-02bbf11bcca1 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:59.845838092 +0000 UTC m=+61.105748495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jw78t" (UID: "ee6ee131-6cea-42c7-80c2-02bbf11bcca1") : secret "samples-operator-tls" not found Jul 16 21:50:59.046631 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:59.046590 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:59.046631 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:59.046635 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:50:59.047127 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:59.046771 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 21:50:59.047127 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:59.046786 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle podName:5db1862a-2a05-4cf2-bc16-05f4d1188f88 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:00.046768028 +0000 UTC m=+61.306678448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle") pod "router-default-c99d8d4d4-4trjg" (UID: "5db1862a-2a05-4cf2-bc16-05f4d1188f88") : configmap references non-existent config key: service-ca.crt Jul 16 21:50:59.047127 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:59.046884 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs podName:5db1862a-2a05-4cf2-bc16-05f4d1188f88 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:00.04683023 +0000 UTC m=+61.306740637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs") pod "router-default-c99d8d4d4-4trjg" (UID: "5db1862a-2a05-4cf2-bc16-05f4d1188f88") : secret "router-metrics-certs-default" not found Jul 16 21:50:59.561966 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:59.561891 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" event={"ID":"5cd0ba43-b852-4efc-bfb9-0303473bb89c","Type":"ContainerStarted","Data":"21ae4802c1cf49482210d01c2e3763646fdfaeb84381e24289eb60b1d6128548"} Jul 16 21:50:59.851075 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:50:59.850988 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jw78t\" (UID: \"ee6ee131-6cea-42c7-80c2-02bbf11bcca1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:50:59.851243 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:59.851158 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 21:50:59.851243 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:50:59.851237 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls podName:ee6ee131-6cea-42c7-80c2-02bbf11bcca1 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:01.851218791 +0000 UTC m=+63.111129193 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jw78t" (UID: "ee6ee131-6cea-42c7-80c2-02bbf11bcca1") : secret "samples-operator-tls" not found Jul 16 21:51:00.052261 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:00.052225 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:00.052261 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:00.052267 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:00.052713 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:00.052381 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 21:51:00.052713 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:00.052465 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle podName:5db1862a-2a05-4cf2-bc16-05f4d1188f88 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:02.052448728 +0000 UTC m=+63.312359153 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle") pod "router-default-c99d8d4d4-4trjg" (UID: "5db1862a-2a05-4cf2-bc16-05f4d1188f88") : configmap references non-existent config key: service-ca.crt Jul 16 21:51:00.052713 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:00.052484 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs podName:5db1862a-2a05-4cf2-bc16-05f4d1188f88 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:02.052477778 +0000 UTC m=+63.312388179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs") pod "router-default-c99d8d4d4-4trjg" (UID: "5db1862a-2a05-4cf2-bc16-05f4d1188f88") : secret "router-metrics-certs-default" not found Jul 16 21:51:00.487065 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:00.487040 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7ttlc" Jul 16 21:51:01.568206 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:01.568160 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" event={"ID":"37690e6c-a753-4329-8d1b-f327f3cf2c25","Type":"ContainerStarted","Data":"22d3b04a0b80bf70efbd1f43a1c1b8a86df3762511ef987e29f17e8063618257"} Jul 16 21:51:01.569689 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:01.569663 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" event={"ID":"5cd0ba43-b852-4efc-bfb9-0303473bb89c","Type":"ContainerStarted","Data":"25f641823085e2317f8d681d24cec5f5535fa4ce24dd7d0567d309d933a9c45c"} Jul 16 21:51:01.637628 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:01.637565 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" podStartSLOduration=1.319515563 podStartE2EDuration="3.637543649s" totalImagesPullingTime="2.318028086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:01.636728049 +0000 UTC m=+62.896638474" watchObservedRunningTime="2026-07-16 21:51:01.637543649 +0000 UTC m=+62.897454051" Jul 16 21:51:01.867100 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:01.867024 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jw78t\" (UID: \"ee6ee131-6cea-42c7-80c2-02bbf11bcca1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:51:01.867286 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:01.867174 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 21:51:01.867286 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:01.867238 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls podName:ee6ee131-6cea-42c7-80c2-02bbf11bcca1 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:05.867221785 +0000 UTC m=+67.127132187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jw78t" (UID: "ee6ee131-6cea-42c7-80c2-02bbf11bcca1") : secret "samples-operator-tls" not found Jul 16 21:51:02.069045 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.069006 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:02.069045 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.069048 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:02.069251 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:02.069144 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 21:51:02.069251 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:02.069170 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle podName:5db1862a-2a05-4cf2-bc16-05f4d1188f88 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:06.069156775 +0000 UTC m=+67.329067177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle") pod "router-default-c99d8d4d4-4trjg" (UID: "5db1862a-2a05-4cf2-bc16-05f4d1188f88") : configmap references non-existent config key: service-ca.crt Jul 16 21:51:02.069251 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:02.069195 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs podName:5db1862a-2a05-4cf2-bc16-05f4d1188f88 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:06.069182861 +0000 UTC m=+67.329093263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs") pod "router-default-c99d8d4d4-4trjg" (UID: "5db1862a-2a05-4cf2-bc16-05f4d1188f88") : secret "router-metrics-certs-default" not found Jul 16 21:51:02.618826 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.618770 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" podStartSLOduration=2.450556429 podStartE2EDuration="4.618748537s" totalImagesPullingTime="2.168192108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:01.680474473 +0000 UTC m=+62.940384897" watchObservedRunningTime="2026-07-16 21:51:02.618748537 +0000 UTC m=+63.878658962" Jul 16 21:51:02.619271 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.618913 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68"] Jul 16 21:51:02.623583 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.623557 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68" Jul 16 21:51:02.627385 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.627363 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 16 21:51:02.627529 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.627513 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tkj6d\"" Jul 16 21:51:02.628141 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.628122 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 16 21:51:02.634492 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.634470 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68"] Jul 16 21:51:02.673771 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.673746 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plhc9\" (UniqueName: \"kubernetes.io/projected/bd52e79a-6250-458a-8547-6c4aa794f7d2-kube-api-access-plhc9\") pod \"migrator-7cc9c7ffd7-brx68\" (UID: \"bd52e79a-6250-458a-8547-6c4aa794f7d2\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68" Jul 16 21:51:02.774958 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.774899 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plhc9\" (UniqueName: \"kubernetes.io/projected/bd52e79a-6250-458a-8547-6c4aa794f7d2-kube-api-access-plhc9\") pod \"migrator-7cc9c7ffd7-brx68\" (UID: \"bd52e79a-6250-458a-8547-6c4aa794f7d2\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68" Jul 16 21:51:02.784308 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.784284 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plhc9\" (UniqueName: \"kubernetes.io/projected/bd52e79a-6250-458a-8547-6c4aa794f7d2-kube-api-access-plhc9\") pod \"migrator-7cc9c7ffd7-brx68\" (UID: \"bd52e79a-6250-458a-8547-6c4aa794f7d2\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68" Jul 16 21:51:02.932816 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:02.932786 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68" Jul 16 21:51:03.075949 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:03.075922 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68"] Jul 16 21:51:03.078776 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:03.078754 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd52e79a_6250_458a_8547_6c4aa794f7d2.slice/crio-cc77522a07b373ec34d90ad78ade80a886e77d8cddd374c4ce9a901d46e7cfb2 WatchSource:0}: Error finding container cc77522a07b373ec34d90ad78ade80a886e77d8cddd374c4ce9a901d46e7cfb2: Status 404 returned error can't find the container with id cc77522a07b373ec34d90ad78ade80a886e77d8cddd374c4ce9a901d46e7cfb2 Jul 16 21:51:03.574735 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:03.574704 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68" event={"ID":"bd52e79a-6250-458a-8547-6c4aa794f7d2","Type":"ContainerStarted","Data":"cc77522a07b373ec34d90ad78ade80a886e77d8cddd374c4ce9a901d46e7cfb2"} Jul 16 21:51:04.081425 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.081392 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-89sfs"] Jul 16 21:51:04.083736 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.083714 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.087213 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.087190 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 21:51:04.087486 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.087468 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 21:51:04.088311 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.088100 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 21:51:04.088311 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.088141 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8tffc\"" Jul 16 21:51:04.089280 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.089264 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 21:51:04.110987 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.110960 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89sfs"] Jul 16 21:51:04.183908 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.183875 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xs7\" (UniqueName: \"kubernetes.io/projected/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-api-access-c9xs7\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.184080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.183927 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/845e715f-d30d-4a40-b41c-edb7671ae0a2-data-volume\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.184080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.183949 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.184080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.184034 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.184080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.184068 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/845e715f-d30d-4a40-b41c-edb7671ae0a2-crio-socket\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.285006 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.284976 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/845e715f-d30d-4a40-b41c-edb7671ae0a2-data-volume\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.285202 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.285016 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.285202 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.285069 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.285202 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.285095 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/845e715f-d30d-4a40-b41c-edb7671ae0a2-crio-socket\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.285202 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:04.285191 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:51:04.285401 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:04.285234 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:51:04.285401 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:04.285265 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm podName:845e715f-d30d-4a40-b41c-edb7671ae0a2 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:04.785243625 +0000 UTC m=+66.045154030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89sfs" (UID: "845e715f-d30d-4a40-b41c-edb7671ae0a2") : configmap "kube-rbac-proxy" not found Jul 16 21:51:04.285401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.285200 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xs7\" (UniqueName: \"kubernetes.io/projected/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-api-access-c9xs7\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.285401 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:04.285285 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls podName:845e715f-d30d-4a40-b41c-edb7671ae0a2 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:04.785273847 +0000 UTC m=+66.045184250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89sfs" (UID: "845e715f-d30d-4a40-b41c-edb7671ae0a2") : secret "insights-runtime-extractor-tls" not found Jul 16 21:51:04.285401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.285293 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/845e715f-d30d-4a40-b41c-edb7671ae0a2-crio-socket\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.285401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.285372 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/845e715f-d30d-4a40-b41c-edb7671ae0a2-data-volume\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.288379 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.288356 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-4zf55"] Jul 16 21:51:04.290074 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.290061 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-4zf55" Jul 16 21:51:04.296742 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.296718 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 16 21:51:04.296911 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.296748 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 16 21:51:04.297242 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.297226 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 16 21:51:04.298693 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.298668 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 16 21:51:04.301630 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.301609 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7thls\"" Jul 16 21:51:04.303714 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.303692 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xs7\" (UniqueName: \"kubernetes.io/projected/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-api-access-c9xs7\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.311761 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.311739 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-4zf55"] Jul 16 21:51:04.385897 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.385796 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz54w\" (UniqueName: \"kubernetes.io/projected/17460c1f-19d7-4afb-84dd-ae963d05d131-kube-api-access-nz54w\") pod \"service-ca-58598695f7-4zf55\" (UID: \"17460c1f-19d7-4afb-84dd-ae963d05d131\") " pod="openshift-service-ca/service-ca-58598695f7-4zf55" Jul 16 21:51:04.386050 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.385915 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/17460c1f-19d7-4afb-84dd-ae963d05d131-signing-key\") pod \"service-ca-58598695f7-4zf55\" (UID: \"17460c1f-19d7-4afb-84dd-ae963d05d131\") " pod="openshift-service-ca/service-ca-58598695f7-4zf55" Jul 16 21:51:04.386050 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.385964 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/17460c1f-19d7-4afb-84dd-ae963d05d131-signing-cabundle\") pod \"service-ca-58598695f7-4zf55\" (UID: \"17460c1f-19d7-4afb-84dd-ae963d05d131\") " pod="openshift-service-ca/service-ca-58598695f7-4zf55" Jul 16 21:51:04.487324 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.487283 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/17460c1f-19d7-4afb-84dd-ae963d05d131-signing-key\") pod \"service-ca-58598695f7-4zf55\" (UID: \"17460c1f-19d7-4afb-84dd-ae963d05d131\") " pod="openshift-service-ca/service-ca-58598695f7-4zf55" Jul 16 21:51:04.487493 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.487342 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/17460c1f-19d7-4afb-84dd-ae963d05d131-signing-cabundle\") pod \"service-ca-58598695f7-4zf55\" (UID: \"17460c1f-19d7-4afb-84dd-ae963d05d131\") " pod="openshift-service-ca/service-ca-58598695f7-4zf55" Jul 16 21:51:04.487493 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.487438 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz54w\" (UniqueName: \"kubernetes.io/projected/17460c1f-19d7-4afb-84dd-ae963d05d131-kube-api-access-nz54w\") pod \"service-ca-58598695f7-4zf55\" (UID: \"17460c1f-19d7-4afb-84dd-ae963d05d131\") " pod="openshift-service-ca/service-ca-58598695f7-4zf55" Jul 16 21:51:04.488185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.488160 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/17460c1f-19d7-4afb-84dd-ae963d05d131-signing-cabundle\") pod \"service-ca-58598695f7-4zf55\" (UID: \"17460c1f-19d7-4afb-84dd-ae963d05d131\") " pod="openshift-service-ca/service-ca-58598695f7-4zf55" Jul 16 21:51:04.490325 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.490302 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/17460c1f-19d7-4afb-84dd-ae963d05d131-signing-key\") pod \"service-ca-58598695f7-4zf55\" (UID: \"17460c1f-19d7-4afb-84dd-ae963d05d131\") " pod="openshift-service-ca/service-ca-58598695f7-4zf55" Jul 16 21:51:04.503527 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.503501 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz54w\" (UniqueName: \"kubernetes.io/projected/17460c1f-19d7-4afb-84dd-ae963d05d131-kube-api-access-nz54w\") pod \"service-ca-58598695f7-4zf55\" (UID: \"17460c1f-19d7-4afb-84dd-ae963d05d131\") " pod="openshift-service-ca/service-ca-58598695f7-4zf55" Jul 16 21:51:04.599093 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.599073 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-4zf55" Jul 16 21:51:04.729373 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.729342 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-4zf55"] Jul 16 21:51:04.732808 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:04.732780 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17460c1f_19d7_4afb_84dd_ae963d05d131.slice/crio-d0b415922f2dd6c6117dbd58c22a8320a7acf9504779c83aea2c2e99047ceb26 WatchSource:0}: Error finding container d0b415922f2dd6c6117dbd58c22a8320a7acf9504779c83aea2c2e99047ceb26: Status 404 returned error can't find the container with id d0b415922f2dd6c6117dbd58c22a8320a7acf9504779c83aea2c2e99047ceb26 Jul 16 21:51:04.789179 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.789154 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.789265 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.789193 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:04.789312 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:04.789276 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:51:04.789346 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:04.789339 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm podName:845e715f-d30d-4a40-b41c-edb7671ae0a2 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:05.789319235 +0000 UTC m=+67.049229642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89sfs" (UID: "845e715f-d30d-4a40-b41c-edb7671ae0a2") : configmap "kube-rbac-proxy" not found Jul 16 21:51:04.789384 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:04.789352 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:51:04.789413 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:04.789384 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls podName:845e715f-d30d-4a40-b41c-edb7671ae0a2 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:05.789376196 +0000 UTC m=+67.049286598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89sfs" (UID: "845e715f-d30d-4a40-b41c-edb7671ae0a2") : secret "insights-runtime-extractor-tls" not found Jul 16 21:51:04.990737 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.990702 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:51:04.990940 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.990782 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:51:04.990940 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.990814 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4kv\" (UniqueName: \"kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv\") pod \"network-check-target-nhjhp\" (UID: \"21d8755c-3957-4432-87d7-312fb00a1d6a\") " pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:51:04.996146 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.996093 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 21:51:04.996263 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.996187 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 21:51:04.996263 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:04.996235 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 21:51:05.001579 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:05.001562 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 21:51:05.001665 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:05.001625 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs podName:7e676b3e-e7cf-41d1-ac8c-e2177080ea66 nodeName:}" failed. No retries permitted until 2026-07-16 21:52:09.001605624 +0000 UTC m=+130.261516032 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs") pod "network-metrics-daemon-wkshg" (UID: "7e676b3e-e7cf-41d1-ac8c-e2177080ea66") : secret "metrics-daemon-secret" not found Jul 16 21:51:05.003880 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.003838 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c1ed479-ddc7-4099-b273-ddd8c98534b4-original-pull-secret\") pod \"global-pull-secret-syncer-jkdtg\" (UID: \"9c1ed479-ddc7-4099-b273-ddd8c98534b4\") " pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:51:05.004419 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.004404 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 21:51:05.014544 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.014527 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4kv\" (UniqueName: \"kubernetes.io/projected/21d8755c-3957-4432-87d7-312fb00a1d6a-kube-api-access-tw4kv\") pod \"network-check-target-nhjhp\" (UID: \"21d8755c-3957-4432-87d7-312fb00a1d6a\") " pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:51:05.067771 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.067752 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jkdtg" Jul 16 21:51:05.081931 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.078170 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqtjl\"" Jul 16 21:51:05.084121 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.084100 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:51:05.228799 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.228770 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jkdtg"] Jul 16 21:51:05.232294 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:05.232268 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1ed479_ddc7_4099_b273_ddd8c98534b4.slice/crio-eb2f233e57b651f86ec4cfe0526175436543db4856fdb695c18dfaf77a44583c WatchSource:0}: Error finding container eb2f233e57b651f86ec4cfe0526175436543db4856fdb695c18dfaf77a44583c: Status 404 returned error can't find the container with id eb2f233e57b651f86ec4cfe0526175436543db4856fdb695c18dfaf77a44583c Jul 16 21:51:05.242054 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.241993 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nhjhp"] Jul 16 21:51:05.245211 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:05.245188 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d8755c_3957_4432_87d7_312fb00a1d6a.slice/crio-eb47340fbc44878b15c2078b7ed3501472543616510e3d6c965afde52334a442 WatchSource:0}: Error finding container eb47340fbc44878b15c2078b7ed3501472543616510e3d6c965afde52334a442: Status 404 returned error can't find the container with id eb47340fbc44878b15c2078b7ed3501472543616510e3d6c965afde52334a442 Jul 16 21:51:05.582997 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.582900 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68" event={"ID":"bd52e79a-6250-458a-8547-6c4aa794f7d2","Type":"ContainerStarted","Data":"0b4114e0fb5d4d4fce30d842f34b7284a1d267bc714116f70909195aec5833d0"} Jul 16 21:51:05.582997 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.582965 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68" event={"ID":"bd52e79a-6250-458a-8547-6c4aa794f7d2","Type":"ContainerStarted","Data":"65e0901da5af3e8f9266f8eeac420ef042b25f70695954f1cd071e6316b0294f"} Jul 16 21:51:05.584100 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.584075 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nhjhp" event={"ID":"21d8755c-3957-4432-87d7-312fb00a1d6a","Type":"ContainerStarted","Data":"eb47340fbc44878b15c2078b7ed3501472543616510e3d6c965afde52334a442"} Jul 16 21:51:05.585178 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.585143 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jkdtg" event={"ID":"9c1ed479-ddc7-4099-b273-ddd8c98534b4","Type":"ContainerStarted","Data":"eb2f233e57b651f86ec4cfe0526175436543db4856fdb695c18dfaf77a44583c"} Jul 16 21:51:05.586447 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.586425 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-4zf55" event={"ID":"17460c1f-19d7-4afb-84dd-ae963d05d131","Type":"ContainerStarted","Data":"9980f8ee3bed267f33b94ca8880c22ed9303557646c80431334b29d95761a879"} Jul 16 21:51:05.586520 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.586465 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-4zf55" event={"ID":"17460c1f-19d7-4afb-84dd-ae963d05d131","Type":"ContainerStarted","Data":"d0b415922f2dd6c6117dbd58c22a8320a7acf9504779c83aea2c2e99047ceb26"} Jul 16 21:51:05.611108 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.611064 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-brx68" podStartSLOduration=2.148227137 podStartE2EDuration="3.611046436s" totalImagesPullingTime="1.462819299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:05.610684217 +0000 UTC m=+66.870594640" watchObservedRunningTime="2026-07-16 21:51:05.611046436 +0000 UTC m=+66.870956863" Jul 16 21:51:05.797101 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.797069 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:05.797271 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.797109 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:05.797271 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:05.797207 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:51:05.797382 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:05.797280 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:51:05.797382 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:05.797282 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm podName:845e715f-d30d-4a40-b41c-edb7671ae0a2 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:07.797263209 +0000 UTC m=+69.057173610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89sfs" (UID: "845e715f-d30d-4a40-b41c-edb7671ae0a2") : configmap "kube-rbac-proxy" not found Jul 16 21:51:05.797382 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:05.797328 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls podName:845e715f-d30d-4a40-b41c-edb7671ae0a2 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:07.797315997 +0000 UTC m=+69.057226400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89sfs" (UID: "845e715f-d30d-4a40-b41c-edb7671ae0a2") : secret "insights-runtime-extractor-tls" not found Jul 16 21:51:05.897986 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:05.897900 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jw78t\" (UID: \"ee6ee131-6cea-42c7-80c2-02bbf11bcca1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:51:05.898127 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:05.898041 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 16 21:51:05.898127 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:05.898111 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls podName:ee6ee131-6cea-42c7-80c2-02bbf11bcca1 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:13.898090392 +0000 UTC m=+75.158000800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-jw78t" (UID: "ee6ee131-6cea-42c7-80c2-02bbf11bcca1") : secret "samples-operator-tls" not found Jul 16 21:51:06.099749 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:06.099710 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:06.100175 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:06.099804 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:06.100175 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:06.099893 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 16 21:51:06.100175 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:06.099951 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs podName:5db1862a-2a05-4cf2-bc16-05f4d1188f88 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:14.099933981 +0000 UTC m=+75.359844383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs") pod "router-default-c99d8d4d4-4trjg" (UID: "5db1862a-2a05-4cf2-bc16-05f4d1188f88") : secret "router-metrics-certs-default" not found Jul 16 21:51:06.100175 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:06.099970 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle podName:5db1862a-2a05-4cf2-bc16-05f4d1188f88 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:14.099960538 +0000 UTC m=+75.359870942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle") pod "router-default-c99d8d4d4-4trjg" (UID: "5db1862a-2a05-4cf2-bc16-05f4d1188f88") : configmap references non-existent config key: service-ca.crt Jul 16 21:51:06.503593 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:06.503552 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:51:06.503776 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:06.503760 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:51:06.503843 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:06.503822 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls podName:3c0c065d-3277-4995-beca-874814b115fa nodeName:}" failed. No retries permitted until 2026-07-16 21:51:38.50380381 +0000 UTC m=+99.763714224 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls") pod "dns-default-xgq9j" (UID: "3c0c065d-3277-4995-beca-874814b115fa") : secret "dns-default-metrics-tls" not found Jul 16 21:51:06.628842 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:06.628817 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b2wz8_c29a9f8e-4098-45a1-8c53-84c0fc20f0c1/dns-node-resolver/0.log" Jul 16 21:51:07.662845 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:07.662733 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8stlx_3347b12c-12e0-4f72-a9eb-9480db6fd042/node-ca/0.log" Jul 16 21:51:07.823443 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:07.822773 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:07.823443 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:07.822821 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:07.823443 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:07.823018 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:51:07.823443 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:07.823082 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls podName:845e715f-d30d-4a40-b41c-edb7671ae0a2 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:11.823063359 +0000 UTC m=+73.082973764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89sfs" (UID: "845e715f-d30d-4a40-b41c-edb7671ae0a2") : secret "insights-runtime-extractor-tls" not found Jul 16 21:51:07.823443 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:07.823218 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:51:07.823443 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:07.823265 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm podName:845e715f-d30d-4a40-b41c-edb7671ae0a2 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:11.823250584 +0000 UTC m=+73.083160986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89sfs" (UID: "845e715f-d30d-4a40-b41c-edb7671ae0a2") : configmap "kube-rbac-proxy" not found Jul 16 21:51:08.436520 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:08.436489 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mz7fk_cad724ee-e0b4-4519-9f88-52e577024ced/serve-healthcheck-canary/0.log" Jul 16 21:51:09.031288 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:09.031228 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-brx68_bd52e79a-6250-458a-8547-6c4aa794f7d2/migrator/0.log" Jul 16 21:51:09.229936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:09.229905 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-brx68_bd52e79a-6250-458a-8547-6c4aa794f7d2/graceful-termination/0.log" Jul 16 21:51:09.431812 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:09.431785 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-s9jq4_5cd0ba43-b852-4efc-bfb9-0303473bb89c/kube-storage-version-migrator-operator/0.log" Jul 16 21:51:09.604629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:09.604594 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nhjhp" event={"ID":"21d8755c-3957-4432-87d7-312fb00a1d6a","Type":"ContainerStarted","Data":"c5ba6dd9b753053fb74fa774df25d4aad799266b59efaca6ba5a9c06cc76d56a"} Jul 16 21:51:09.604818 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:09.604747 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:51:09.627196 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:09.626290 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nhjhp" podStartSLOduration=65.732518776 podStartE2EDuration="1m9.626271047s" totalImagesPullingTime="3.893752271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:09.625301981 +0000 UTC m=+70.885212428" watchObservedRunningTime="2026-07-16 21:51:09.626271047 +0000 UTC m=+70.886181472" Jul 16 21:51:09.627196 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:09.626637 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-4zf55" podStartSLOduration=5.626627278 podStartE2EDuration="5.626627278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:05.635418178 +0000 UTC m=+66.895328601" watchObservedRunningTime="2026-07-16 21:51:09.626627278 +0000 UTC m=+70.886537704" Jul 16 21:51:11.613336 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:11.613296 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jkdtg" event={"ID":"9c1ed479-ddc7-4099-b273-ddd8c98534b4","Type":"ContainerStarted","Data":"d0ec988dc18298ddb9739404fd3d8bf77db0f3946cf2f1de8f6e0a735a82d994"} Jul 16 21:51:11.633577 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:11.633530 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jkdtg" podStartSLOduration=66.081102 podStartE2EDuration="1m11.63351525s" totalImagesPullingTime="5.55241325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:11.632458859 +0000 UTC m=+72.892369282" watchObservedRunningTime="2026-07-16 21:51:11.63351525 +0000 UTC m=+72.893425719" Jul 16 21:51:11.858478 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:11.858439 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:11.858478 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:11.858477 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:11.858722 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:11.858562 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:51:11.858722 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:11.858610 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:51:11.858722 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:11.858634 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm podName:845e715f-d30d-4a40-b41c-edb7671ae0a2 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:19.858614713 +0000 UTC m=+81.118525115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-89sfs" (UID: "845e715f-d30d-4a40-b41c-edb7671ae0a2") : configmap "kube-rbac-proxy" not found Jul 16 21:51:11.858722 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:51:11.858651 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls podName:845e715f-d30d-4a40-b41c-edb7671ae0a2 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:19.858642185 +0000 UTC m=+81.118552589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-89sfs" (UID: "845e715f-d30d-4a40-b41c-edb7671ae0a2") : secret "insights-runtime-extractor-tls" not found Jul 16 21:51:13.974088 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:13.974054 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jw78t\" (UID: \"ee6ee131-6cea-42c7-80c2-02bbf11bcca1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:51:13.976569 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:13.976538 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ee6ee131-6cea-42c7-80c2-02bbf11bcca1-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-jw78t\" (UID: \"ee6ee131-6cea-42c7-80c2-02bbf11bcca1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:51:14.050421 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.050390 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tgxzg\"" Jul 16 21:51:14.058278 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.058262 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" Jul 16 21:51:14.175989 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.175958 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:14.176151 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.176027 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:14.176560 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.176540 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5db1862a-2a05-4cf2-bc16-05f4d1188f88-service-ca-bundle\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:14.178395 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.178377 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5db1862a-2a05-4cf2-bc16-05f4d1188f88-metrics-certs\") pod \"router-default-c99d8d4d4-4trjg\" (UID: \"5db1862a-2a05-4cf2-bc16-05f4d1188f88\") " pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:14.233493 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.233361 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t"] Jul 16 21:51:14.467693 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.467662 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gcvjv\"" Jul 16 21:51:14.475796 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.475777 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:14.615084 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.615059 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c99d8d4d4-4trjg"] Jul 16 21:51:14.617496 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:14.617471 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db1862a_2a05_4cf2_bc16_05f4d1188f88.slice/crio-fc8cda4d76e5c747a269321acf9e6d9cd579563af2e223a3e07126c8934f7839 WatchSource:0}: Error finding container fc8cda4d76e5c747a269321acf9e6d9cd579563af2e223a3e07126c8934f7839: Status 404 returned error can't find the container with id fc8cda4d76e5c747a269321acf9e6d9cd579563af2e223a3e07126c8934f7839 Jul 16 21:51:14.623035 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.623006 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c99d8d4d4-4trjg" event={"ID":"5db1862a-2a05-4cf2-bc16-05f4d1188f88","Type":"ContainerStarted","Data":"fc8cda4d76e5c747a269321acf9e6d9cd579563af2e223a3e07126c8934f7839"} Jul 16 21:51:14.624145 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:14.624124 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" event={"ID":"ee6ee131-6cea-42c7-80c2-02bbf11bcca1","Type":"ContainerStarted","Data":"d100796418625aa5ed6052387c3c5bc60756880aeb2eaad02fbb6d879e1d0952"} Jul 16 21:51:15.631124 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:15.631089 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c99d8d4d4-4trjg" event={"ID":"5db1862a-2a05-4cf2-bc16-05f4d1188f88","Type":"ContainerStarted","Data":"79e4dd54e9415f28df39a8b9bd95f87b1b6174d8530fe8e9dc9f4593137e3f7d"} Jul 16 21:51:15.656999 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:15.656946 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-c99d8d4d4-4trjg" podStartSLOduration=17.656927167 podStartE2EDuration="17.656927167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:15.65565222 +0000 UTC m=+76.915562645" watchObservedRunningTime="2026-07-16 21:51:15.656927167 +0000 UTC m=+76.916837585" Jul 16 21:51:16.476325 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:16.476291 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:16.478957 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:16.478934 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:16.636184 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:16.636149 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" event={"ID":"ee6ee131-6cea-42c7-80c2-02bbf11bcca1","Type":"ContainerStarted","Data":"77fcde8ef47241265c4c0a7251c65d71bb0268686f1dd879a4fe25c8d90c43fa"} Jul 16 21:51:16.636624 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:16.636360 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:16.636624 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:16.636404 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" event={"ID":"ee6ee131-6cea-42c7-80c2-02bbf11bcca1","Type":"ContainerStarted","Data":"ee5b35607425d3e6ca6d9f2c6de8cff3677f78dc0a56ff21e332bdd86c8ac8dc"} Jul 16 21:51:16.637525 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:16.637504 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c99d8d4d4-4trjg" Jul 16 21:51:16.679918 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:16.679851 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-jw78t" podStartSLOduration=17.13700552 podStartE2EDuration="18.679835632s" totalImagesPullingTime="1.542830112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:16.67880979 +0000 UTC m=+77.938720234" watchObservedRunningTime="2026-07-16 21:51:16.679835632 +0000 UTC m=+77.939746077" Jul 16 21:51:19.922183 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:19.922133 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:19.922183 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:19.922186 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:19.922772 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:19.922744 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/845e715f-d30d-4a40-b41c-edb7671ae0a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:19.924631 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:19.924610 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/845e715f-d30d-4a40-b41c-edb7671ae0a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89sfs\" (UID: \"845e715f-d30d-4a40-b41c-edb7671ae0a2\") " pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:19.994623 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:19.994586 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89sfs" Jul 16 21:51:20.123180 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:20.123033 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89sfs"] Jul 16 21:51:20.128071 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:20.128044 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845e715f_d30d_4a40_b41c_edb7671ae0a2.slice/crio-1c1bace0ed711c32b4bb9cce2882df4dde0bc94e30d67f514f7c3677cdfa279d WatchSource:0}: Error finding container 1c1bace0ed711c32b4bb9cce2882df4dde0bc94e30d67f514f7c3677cdfa279d: Status 404 returned error can't find the container with id 1c1bace0ed711c32b4bb9cce2882df4dde0bc94e30d67f514f7c3677cdfa279d Jul 16 21:51:20.654713 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:20.654630 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89sfs" event={"ID":"845e715f-d30d-4a40-b41c-edb7671ae0a2","Type":"ContainerStarted","Data":"1c1bace0ed711c32b4bb9cce2882df4dde0bc94e30d67f514f7c3677cdfa279d"} Jul 16 21:51:25.673656 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:25.673618 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89sfs" event={"ID":"845e715f-d30d-4a40-b41c-edb7671ae0a2","Type":"ContainerStarted","Data":"5922e0009159f36168143ee59ecfc519a9bdefb93dc3b554d0176d0307c9225c"} Jul 16 21:51:28.685683 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:28.685651 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89sfs" event={"ID":"845e715f-d30d-4a40-b41c-edb7671ae0a2","Type":"ContainerStarted","Data":"14a8ee802ab38985075ee3ff4cdbb78932721dba975ffcfc39aac498171eb9e7"} Jul 16 21:51:30.731610 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.731574 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-964956bf6-6tdrb"] Jul 16 21:51:30.742193 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.742162 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.744701 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.744676 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 21:51:30.744824 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.744677 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nmm9q\"" Jul 16 21:51:30.745111 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.745091 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 21:51:30.746034 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.746018 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 21:51:30.750228 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.749382 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-964956bf6-6tdrb"] Jul 16 21:51:30.751264 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.751244 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 21:51:30.814974 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.814885 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78e10644-eb84-4ed1-9100-439190cd5bec-image-registry-private-configuration\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.814974 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.814937 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78e10644-eb84-4ed1-9100-439190cd5bec-registry-certificates\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.814974 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.814974 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78e10644-eb84-4ed1-9100-439190cd5bec-bound-sa-token\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.815225 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.815077 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l7jq\" (UniqueName: \"kubernetes.io/projected/78e10644-eb84-4ed1-9100-439190cd5bec-kube-api-access-8l7jq\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.815225 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.815121 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78e10644-eb84-4ed1-9100-439190cd5bec-installation-pull-secrets\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.815225 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.815172 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78e10644-eb84-4ed1-9100-439190cd5bec-registry-tls\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.815339 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.815237 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78e10644-eb84-4ed1-9100-439190cd5bec-trusted-ca\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.815339 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.815279 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78e10644-eb84-4ed1-9100-439190cd5bec-ca-trust-extracted\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.916357 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.916321 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78e10644-eb84-4ed1-9100-439190cd5bec-trusted-ca\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.916523 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.916379 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78e10644-eb84-4ed1-9100-439190cd5bec-ca-trust-extracted\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.916523 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.916412 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78e10644-eb84-4ed1-9100-439190cd5bec-image-registry-private-configuration\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.916523 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.916438 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78e10644-eb84-4ed1-9100-439190cd5bec-registry-certificates\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.916523 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.916474 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78e10644-eb84-4ed1-9100-439190cd5bec-bound-sa-token\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.916741 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.916536 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l7jq\" (UniqueName: \"kubernetes.io/projected/78e10644-eb84-4ed1-9100-439190cd5bec-kube-api-access-8l7jq\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.916741 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.916563 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78e10644-eb84-4ed1-9100-439190cd5bec-installation-pull-secrets\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.916741 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.916610 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78e10644-eb84-4ed1-9100-439190cd5bec-registry-tls\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.916912 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.916837 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/78e10644-eb84-4ed1-9100-439190cd5bec-ca-trust-extracted\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.917449 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.917413 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78e10644-eb84-4ed1-9100-439190cd5bec-trusted-ca\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.917875 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.917837 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/78e10644-eb84-4ed1-9100-439190cd5bec-registry-certificates\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.919410 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.919385 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/78e10644-eb84-4ed1-9100-439190cd5bec-image-registry-private-configuration\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.919652 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.919629 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/78e10644-eb84-4ed1-9100-439190cd5bec-registry-tls\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.919744 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.919700 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/78e10644-eb84-4ed1-9100-439190cd5bec-installation-pull-secrets\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.927167 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.927148 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l7jq\" (UniqueName: \"kubernetes.io/projected/78e10644-eb84-4ed1-9100-439190cd5bec-kube-api-access-8l7jq\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:30.930365 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:30.930345 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78e10644-eb84-4ed1-9100-439190cd5bec-bound-sa-token\") pod \"image-registry-964956bf6-6tdrb\" (UID: \"78e10644-eb84-4ed1-9100-439190cd5bec\") " pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:31.053840 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:31.053796 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:32.045462 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:32.045439 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-964956bf6-6tdrb"] Jul 16 21:51:32.048467 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:32.048443 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e10644_eb84_4ed1_9100_439190cd5bec.slice/crio-5ce1e3f5e6a553c4a00d7a3ed870141134b04f347d6d21ab4b63965cfd0e2f94 WatchSource:0}: Error finding container 5ce1e3f5e6a553c4a00d7a3ed870141134b04f347d6d21ab4b63965cfd0e2f94: Status 404 returned error can't find the container with id 5ce1e3f5e6a553c4a00d7a3ed870141134b04f347d6d21ab4b63965cfd0e2f94 Jul 16 21:51:32.707021 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:32.706982 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89sfs" event={"ID":"845e715f-d30d-4a40-b41c-edb7671ae0a2","Type":"ContainerStarted","Data":"efb9472b2824031cea5f8e276cc947b4e7f90bb2dfb414b39096f4ed84017e94"} Jul 16 21:51:32.708378 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:32.708356 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-964956bf6-6tdrb" event={"ID":"78e10644-eb84-4ed1-9100-439190cd5bec","Type":"ContainerStarted","Data":"2bfd7ded76c41acbf298f29943197b2dc018b4bea09b79dd2c5ca121ca2ba43c"} Jul 16 21:51:32.708519 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:32.708399 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-964956bf6-6tdrb" event={"ID":"78e10644-eb84-4ed1-9100-439190cd5bec","Type":"ContainerStarted","Data":"5ce1e3f5e6a553c4a00d7a3ed870141134b04f347d6d21ab4b63965cfd0e2f94"} Jul 16 21:51:32.708519 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:32.708437 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:32.730172 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:32.730135 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-89sfs" podStartSLOduration=16.997387894 podStartE2EDuration="28.730122291s" totalImagesPullingTime="11.732734397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:32.729164681 +0000 UTC m=+93.989075105" watchObservedRunningTime="2026-07-16 21:51:32.730122291 +0000 UTC m=+93.990032714" Jul 16 21:51:32.756845 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:32.756805 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-964956bf6-6tdrb" podStartSLOduration=2.756791916 podStartE2EDuration="2.756791916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:32.755137289 +0000 UTC m=+94.015047723" watchObservedRunningTime="2026-07-16 21:51:32.756791916 +0000 UTC m=+94.016702339" Jul 16 21:51:38.581673 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:38.581644 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:51:38.583888 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:38.583850 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c0c065d-3277-4995-beca-874814b115fa-metrics-tls\") pod \"dns-default-xgq9j\" (UID: \"3c0c065d-3277-4995-beca-874814b115fa\") " pod="openshift-dns/dns-default-xgq9j" Jul 16 21:51:38.856069 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:38.855992 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6n6rt\"" Jul 16 21:51:38.864321 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:38.864304 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xgq9j" Jul 16 21:51:38.991247 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:38.991203 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xgq9j"] Jul 16 21:51:38.993173 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:38.993148 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0c065d_3277_4995_beca_874814b115fa.slice/crio-a11e5401a75567b37440931eaf3352f233a6a212eefca75ad5aa123e691809e5 WatchSource:0}: Error finding container a11e5401a75567b37440931eaf3352f233a6a212eefca75ad5aa123e691809e5: Status 404 returned error can't find the container with id a11e5401a75567b37440931eaf3352f233a6a212eefca75ad5aa123e691809e5 Jul 16 21:51:39.734271 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:39.734235 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xgq9j" event={"ID":"3c0c065d-3277-4995-beca-874814b115fa","Type":"ContainerStarted","Data":"a11e5401a75567b37440931eaf3352f233a6a212eefca75ad5aa123e691809e5"} Jul 16 21:51:40.529393 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.529357 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q"] Jul 16 21:51:40.531933 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.531910 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.538106 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.538073 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 21:51:40.538834 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.538803 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 21:51:40.538947 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.538830 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 21:51:40.541845 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.539189 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 21:51:40.541845 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.539213 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-g77n2\"" Jul 16 21:51:40.541845 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.539222 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 21:51:40.572623 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.572601 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q"] Jul 16 21:51:40.611588 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.611569 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nhjhp" Jul 16 21:51:40.633455 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.633212 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7wwmt"] Jul 16 21:51:40.637026 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.637003 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.641517 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.641495 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 21:51:40.645620 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.644633 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 21:51:40.645620 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.644763 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-drdzb\"" Jul 16 21:51:40.645620 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.644933 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 21:51:40.696164 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.696136 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a745b7c-fdb1-401b-8ebb-5770258e249e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.696296 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.696196 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8ptp\" (UniqueName: \"kubernetes.io/projected/6a745b7c-fdb1-401b-8ebb-5770258e249e-kube-api-access-s8ptp\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.696365 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.696310 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a745b7c-fdb1-401b-8ebb-5770258e249e-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.696365 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.696341 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a745b7c-fdb1-401b-8ebb-5770258e249e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.740047 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.740022 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xgq9j" event={"ID":"3c0c065d-3277-4995-beca-874814b115fa","Type":"ContainerStarted","Data":"f37d44a98b811b1da50301aad8bcf3e2f0f82895a9aad8978131906d288e941d"} Jul 16 21:51:40.797291 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797229 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a745b7c-fdb1-401b-8ebb-5770258e249e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.797291 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797264 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a745b7c-fdb1-401b-8ebb-5770258e249e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.797291 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797287 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7jqg\" (UniqueName: \"kubernetes.io/projected/5aead329-3038-4b94-b6ae-6879d73fb7b7-kube-api-access-z7jqg\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.797490 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797308 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-textfile\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.797490 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797331 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ptp\" (UniqueName: \"kubernetes.io/projected/6a745b7c-fdb1-401b-8ebb-5770258e249e-kube-api-access-s8ptp\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.797490 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797362 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.797490 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797386 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5aead329-3038-4b94-b6ae-6879d73fb7b7-root\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.797490 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797468 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.797680 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797519 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5aead329-3038-4b94-b6ae-6879d73fb7b7-metrics-client-ca\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.797680 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797594 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aead329-3038-4b94-b6ae-6879d73fb7b7-sys\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.797680 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797620 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-tls\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.797680 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797643 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-wtmp\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.797680 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.797672 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a745b7c-fdb1-401b-8ebb-5770258e249e-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.798420 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.798400 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a745b7c-fdb1-401b-8ebb-5770258e249e-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.799825 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.799804 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a745b7c-fdb1-401b-8ebb-5770258e249e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.800049 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.800029 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a745b7c-fdb1-401b-8ebb-5770258e249e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.810156 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.810137 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ptp\" (UniqueName: \"kubernetes.io/projected/6a745b7c-fdb1-401b-8ebb-5770258e249e-kube-api-access-s8ptp\") pod \"openshift-state-metrics-79f6958cdf-n5l8q\" (UID: \"6a745b7c-fdb1-401b-8ebb-5770258e249e\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.849099 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.849075 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" Jul 16 21:51:40.898210 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898128 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aead329-3038-4b94-b6ae-6879d73fb7b7-sys\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.898210 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898174 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-tls\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.898210 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898200 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-wtmp\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.898440 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898233 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5aead329-3038-4b94-b6ae-6879d73fb7b7-sys\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.898440 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898251 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7jqg\" (UniqueName: \"kubernetes.io/projected/5aead329-3038-4b94-b6ae-6879d73fb7b7-kube-api-access-z7jqg\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.898440 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898289 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-textfile\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.898440 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898326 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.898440 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898352 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5aead329-3038-4b94-b6ae-6879d73fb7b7-root\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.898440 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898384 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.898440 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898420 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5aead329-3038-4b94-b6ae-6879d73fb7b7-metrics-client-ca\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.899465 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898832 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-textfile\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.899465 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898842 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5aead329-3038-4b94-b6ae-6879d73fb7b7-root\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.899465 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.898992 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-wtmp\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.899465 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.899128 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5aead329-3038-4b94-b6ae-6879d73fb7b7-metrics-client-ca\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.899465 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.899410 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.902505 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.901047 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-tls\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.902505 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.901420 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5aead329-3038-4b94-b6ae-6879d73fb7b7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.915710 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.915685 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7jqg\" (UniqueName: \"kubernetes.io/projected/5aead329-3038-4b94-b6ae-6879d73fb7b7-kube-api-access-z7jqg\") pod \"node-exporter-7wwmt\" (UID: \"5aead329-3038-4b94-b6ae-6879d73fb7b7\") " pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.949180 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.949156 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7wwmt" Jul 16 21:51:40.959291 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:40.959268 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aead329_3038_4b94_b6ae_6879d73fb7b7.slice/crio-77c148eaebd2e433ead6e281c55ceb70185ce979ca856cd6996cd5d2b6c105c3 WatchSource:0}: Error finding container 77c148eaebd2e433ead6e281c55ceb70185ce979ca856cd6996cd5d2b6c105c3: Status 404 returned error can't find the container with id 77c148eaebd2e433ead6e281c55ceb70185ce979ca856cd6996cd5d2b6c105c3 Jul 16 21:51:40.979590 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:40.979566 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q"] Jul 16 21:51:40.982384 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:40.982362 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a745b7c_fdb1_401b_8ebb_5770258e249e.slice/crio-5ad8c421bab76b658f1d153e025cdfec7a51937b13a02cd84b94e269365d9f64 WatchSource:0}: Error finding container 5ad8c421bab76b658f1d153e025cdfec7a51937b13a02cd84b94e269365d9f64: Status 404 returned error can't find the container with id 5ad8c421bab76b658f1d153e025cdfec7a51937b13a02cd84b94e269365d9f64 Jul 16 21:51:41.713561 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.713529 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:51:41.727735 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.727713 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.732419 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.732385 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 21:51:41.732699 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.732683 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 21:51:41.732903 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.732888 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 21:51:41.733077 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.733064 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 21:51:41.733326 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.733313 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-44bwl\"" Jul 16 21:51:41.733496 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.733482 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 21:51:41.733690 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.733675 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 21:51:41.733794 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.733775 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 21:51:41.733898 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.733848 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 21:51:41.743014 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.742997 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 21:51:41.746328 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.746301 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wwmt" event={"ID":"5aead329-3038-4b94-b6ae-6879d73fb7b7","Type":"ContainerStarted","Data":"77c148eaebd2e433ead6e281c55ceb70185ce979ca856cd6996cd5d2b6c105c3"} Jul 16 21:51:41.749101 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.749076 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xgq9j" event={"ID":"3c0c065d-3277-4995-beca-874814b115fa","Type":"ContainerStarted","Data":"f2ff6aca6ae68e58e0aa67736b73a862b57ec6ea37b5122c405dfce45daad5a7"} Jul 16 21:51:41.749212 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.749145 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xgq9j" Jul 16 21:51:41.750576 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.750556 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:51:41.753020 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.752885 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" event={"ID":"6a745b7c-fdb1-401b-8ebb-5770258e249e","Type":"ContainerStarted","Data":"8b11aa7e7314c449e1f11a96e52404daa80c85521da4f51637803a3301a56b05"} Jul 16 21:51:41.753020 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.752949 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" event={"ID":"6a745b7c-fdb1-401b-8ebb-5770258e249e","Type":"ContainerStarted","Data":"8dd7e68f071d5dfb5a93d898d0ac81994cb99f2289e9fd5151e7e720402877d4"} Jul 16 21:51:41.753020 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.752988 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" event={"ID":"6a745b7c-fdb1-401b-8ebb-5770258e249e","Type":"ContainerStarted","Data":"5ad8c421bab76b658f1d153e025cdfec7a51937b13a02cd84b94e269365d9f64"} Jul 16 21:51:41.820622 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.820564 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xgq9j" podStartSLOduration=66.211918958 podStartE2EDuration="1m7.820545152s" totalImagesPullingTime="1.608626194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:41.81970411 +0000 UTC m=+103.079614533" watchObservedRunningTime="2026-07-16 21:51:41.820545152 +0000 UTC m=+103.080455579" Jul 16 21:51:41.906103 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906072 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906274 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906123 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-config-volume\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906274 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906179 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906274 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906256 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906407 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906285 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906407 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906339 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906482 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906417 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906482 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906455 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906570 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906498 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906570 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906535 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-config-out\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906661 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906596 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwckm\" (UniqueName: \"kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-kube-api-access-fwckm\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906661 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906632 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:41.906757 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:41.906659 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-web-config\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.007873 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.007771 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.007873 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.007830 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.008080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.007880 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.008080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.007918 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-config-out\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.008080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.007950 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwckm\" (UniqueName: \"kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-kube-api-access-fwckm\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.008080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.007977 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.008080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.008001 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-web-config\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.008080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.008034 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.008080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.008077 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-config-volume\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.008443 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.008100 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.008443 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.008124 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.008443 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.008147 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.008443 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.008197 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.009513 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.009484 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.009697 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.009649 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.009975 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.009696 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.011625 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.011602 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.011725 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.011601 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-config-out\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.012143 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.012103 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-config-volume\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.012287 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.012261 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.012813 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.012770 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-web-config\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.012813 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.012787 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-tls-assets\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.012996 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.012911 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.014198 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.014174 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.014342 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.014324 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.020401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.020376 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwckm\" (UniqueName: \"kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-kube-api-access-fwckm\") pod \"alertmanager-main-0\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.041291 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.041267 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:51:42.411540 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.411515 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:51:42.586679 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:42.586621 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e58536_86af_4312_9a4c_5640b3fddd76.slice/crio-60e702ccc973ed16ba1833544be86abec6328fa269424a9923c4c523b40e03c9 WatchSource:0}: Error finding container 60e702ccc973ed16ba1833544be86abec6328fa269424a9923c4c523b40e03c9: Status 404 returned error can't find the container with id 60e702ccc973ed16ba1833544be86abec6328fa269424a9923c4c523b40e03c9 Jul 16 21:51:42.759246 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:42.759205 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerStarted","Data":"60e702ccc973ed16ba1833544be86abec6328fa269424a9923c4c523b40e03c9"} Jul 16 21:51:43.765328 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:43.765242 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" event={"ID":"6a745b7c-fdb1-401b-8ebb-5770258e249e","Type":"ContainerStarted","Data":"af2759c21fdb5f5ed37517b71a3358b8061df5699149a51cd35934144369a6ce"} Jul 16 21:51:43.767051 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:43.767029 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wwmt" event={"ID":"5aead329-3038-4b94-b6ae-6879d73fb7b7","Type":"ContainerStarted","Data":"970d8181cef380cb9cd9a02a5f96764b6bbb1a067485cb6efa3c3ace9da067b7"} Jul 16 21:51:43.786202 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:43.786161 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-n5l8q" podStartSLOduration=1.971771447 podStartE2EDuration="3.786145976s" totalImagesPullingTime="1.814374529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:43.785028183 +0000 UTC m=+105.044938628" watchObservedRunningTime="2026-07-16 21:51:43.786145976 +0000 UTC m=+105.046056399" Jul 16 21:51:44.775400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:44.775369 2558 generic.go:356] "Generic (PLEG): container finished" podID="97e58536-86af-4312-9a4c-5640b3fddd76" containerID="359246d55f7249d959ba340b907fa44c55afd95168b1a044893c455045b2991d" exitCode=0 Jul 16 21:51:44.775913 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:44.775455 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerDied","Data":"359246d55f7249d959ba340b907fa44c55afd95168b1a044893c455045b2991d"} Jul 16 21:51:44.777071 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:44.777022 2558 generic.go:356] "Generic (PLEG): container finished" podID="5aead329-3038-4b94-b6ae-6879d73fb7b7" containerID="970d8181cef380cb9cd9a02a5f96764b6bbb1a067485cb6efa3c3ace9da067b7" exitCode=0 Jul 16 21:51:44.777171 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:44.777153 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wwmt" event={"ID":"5aead329-3038-4b94-b6ae-6879d73fb7b7","Type":"ContainerDied","Data":"970d8181cef380cb9cd9a02a5f96764b6bbb1a067485cb6efa3c3ace9da067b7"} Jul 16 21:51:45.784361 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:45.784330 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wwmt" event={"ID":"5aead329-3038-4b94-b6ae-6879d73fb7b7","Type":"ContainerStarted","Data":"aab8de0fb30e59288a7fda2edeb7825cbf97aa92edba36d04f33825fc6bcfb75"} Jul 16 21:51:45.784805 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:45.784407 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wwmt" event={"ID":"5aead329-3038-4b94-b6ae-6879d73fb7b7","Type":"ContainerStarted","Data":"486d20f9869d515b4bfd462ce9471a935938e8ff6154c4ba875b240e2fe1573f"} Jul 16 21:51:45.814319 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:45.814264 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7wwmt" podStartSLOduration=3.446593401 podStartE2EDuration="5.814248973s" totalImagesPullingTime="2.263743529s" totalInitContainerRuntime="103.912043ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:45.8132434 +0000 UTC m=+107.073153824" watchObservedRunningTime="2026-07-16 21:51:45.814248973 +0000 UTC m=+107.074159410" Jul 16 21:51:46.796478 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.796440 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerStarted","Data":"062a5221c771a8efda1ee896f3689de148dc885afe2248e61732df757e872f0f"} Jul 16 21:51:46.796960 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.796547 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerStarted","Data":"d1ed0587881083b7734ba99e11750fbcf1bcedce26407b07a6fe862704607b73"} Jul 16 21:51:46.796960 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.796590 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerStarted","Data":"aa0cb5156c6b401fccd635b42460e082d4e1f876e1b5cee16a700342993c29dc"} Jul 16 21:51:46.796960 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.796640 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerStarted","Data":"b4d7c55aa2e953ce4cfcc77334cad77632e8e6133c07582ecb3751bc5647360d"} Jul 16 21:51:46.796960 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.796678 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerStarted","Data":"f86fa0896876456d5fcc33fedef9e47ab26b1f1ab1a74d747faaca1c3e3c590c"} Jul 16 21:51:46.939083 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.939051 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:51:46.941995 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.941975 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:46.952036 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.951621 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 21:51:46.952036 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.951769 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 21:51:46.952036 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.951890 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 21:51:46.952036 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.951898 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 21:51:46.952036 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.951913 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 21:51:46.952036 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.951987 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 21:51:46.952036 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.952002 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 21:51:46.952526 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.952159 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 21:51:46.952526 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.952223 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-21nail2o0jl7a\"" Jul 16 21:51:46.952526 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.952227 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 21:51:46.952526 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.952295 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 21:51:46.952756 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.952563 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 21:51:46.952756 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.952573 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 21:51:46.952756 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.952674 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f92rk\"" Jul 16 21:51:46.952756 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.952724 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 21:51:46.954238 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.954218 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 21:51:46.959000 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.958979 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 21:51:46.965169 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:46.965152 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:51:47.052655 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.052635 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.052725 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.052674 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.052725 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.052697 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.052791 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.052721 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.052791 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.052777 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.052850 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.052822 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053026 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053006 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-config-out\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053059 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053046 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053111 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053096 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053141 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053131 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053173 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053155 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053207 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053177 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053207 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053199 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053266 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053254 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-config\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053297 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053274 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053297 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053290 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053356 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053305 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxw2n\" (UniqueName: \"kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-kube-api-access-hxw2n\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053356 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053327 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053356 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053349 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-web-config\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.053443 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.053373 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154441 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154417 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-config\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154585 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154448 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154585 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154473 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154585 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154488 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxw2n\" (UniqueName: \"kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-kube-api-access-hxw2n\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154585 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154509 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154585 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154536 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-web-config\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154585 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154568 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154594 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154631 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154659 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154691 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154716 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154737 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154779 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-config-out\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154805 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154892 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.154936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154926 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.155581 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154951 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.155581 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.154978 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.155581 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.155006 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.155734 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.155714 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.156267 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.156234 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.156542 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.156281 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.156608 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.156550 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.157038 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.156968 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.158468 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.157645 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.158468 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.157900 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.158468 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.158067 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.158468 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.158395 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.158777 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.158757 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-config-out\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.159056 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.159036 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.159959 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.159821 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.159959 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.159931 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.159959 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.159948 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.160400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.160376 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.160473 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.160414 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-web-config\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.161267 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.161243 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-config\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.161371 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.161354 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.161574 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.161557 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.184565 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.184548 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxw2n\" (UniqueName: \"kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-kube-api-access-hxw2n\") pod \"prometheus-k8s-0\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.252571 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.252546 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:51:47.414477 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.414446 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:51:47.417941 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:51:47.417902 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc445dc85_473b_44dd_86fa_95df14dbf67f.slice/crio-95f9078f53961e6dbfebe30f4b9cbccbe3f9ade550d41420ecb18a729c6652b4 WatchSource:0}: Error finding container 95f9078f53961e6dbfebe30f4b9cbccbe3f9ade550d41420ecb18a729c6652b4: Status 404 returned error can't find the container with id 95f9078f53961e6dbfebe30f4b9cbccbe3f9ade550d41420ecb18a729c6652b4 Jul 16 21:51:47.803599 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.803566 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerStarted","Data":"1ed1a513d7b5eaf345c158ea7b468aa3bf959f07ec0a99c688c38907970985ba"} Jul 16 21:51:47.805150 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.805126 2558 generic.go:356] "Generic (PLEG): container finished" podID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerID="19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a" exitCode=0 Jul 16 21:51:47.805261 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.805178 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerDied","Data":"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a"} Jul 16 21:51:47.805261 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.805241 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerStarted","Data":"95f9078f53961e6dbfebe30f4b9cbccbe3f9ade550d41420ecb18a729c6652b4"} Jul 16 21:51:47.850656 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:47.850602 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.39402528 podStartE2EDuration="6.850583282s" totalImagesPullingTime="3.439007078s" totalInitContainerRuntime="17.550924ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:47.848591785 +0000 UTC m=+109.108502213" watchObservedRunningTime="2026-07-16 21:51:47.850583282 +0000 UTC m=+109.110493707" Jul 16 21:51:50.819889 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:50.819838 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerStarted","Data":"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0"} Jul 16 21:51:50.820230 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:50.819949 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerStarted","Data":"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03"} Jul 16 21:51:51.058438 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:51.058404 2558 patch_prober.go:28] interesting pod/image-registry-964956bf6-6tdrb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 16 21:51:51.058588 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:51.058456 2558 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-964956bf6-6tdrb" podUID="78e10644-eb84-4ed1-9100-439190cd5bec" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 21:51:51.762037 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:51.762004 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xgq9j" Jul 16 21:51:52.832458 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:52.832377 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerStarted","Data":"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d"} Jul 16 21:51:52.832458 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:52.832440 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerStarted","Data":"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870"} Jul 16 21:51:52.832825 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:52.832467 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerStarted","Data":"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237"} Jul 16 21:51:52.832825 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:52.832492 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerStarted","Data":"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa"} Jul 16 21:51:52.866295 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:52.866239 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.336677386 podStartE2EDuration="6.866222575s" totalImagesPullingTime="4.511364283s" totalInitContainerRuntime="18.180906ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:52.865879604 +0000 UTC m=+114.125790021" watchObservedRunningTime="2026-07-16 21:51:52.866222575 +0000 UTC m=+114.126133000" Jul 16 21:51:53.717568 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:53.717538 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-964956bf6-6tdrb" Jul 16 21:51:57.253430 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:51:57.253401 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:52:09.050618 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:09.050582 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:52:09.053158 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:09.053129 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e676b3e-e7cf-41d1-ac8c-e2177080ea66-metrics-certs\") pod \"network-metrics-daemon-wkshg\" (UID: \"7e676b3e-e7cf-41d1-ac8c-e2177080ea66\") " pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:52:09.170690 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:09.170664 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xshkd\"" Jul 16 21:52:09.177672 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:09.177656 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wkshg" Jul 16 21:52:09.376775 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:09.376748 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wkshg"] Jul 16 21:52:09.379067 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:52:09.379038 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e676b3e_e7cf_41d1_ac8c_e2177080ea66.slice/crio-f20b2a1b26270b93cb023c1d9af702d7d1ebf34f65a9b8c156bded52a0c77fa0 WatchSource:0}: Error finding container f20b2a1b26270b93cb023c1d9af702d7d1ebf34f65a9b8c156bded52a0c77fa0: Status 404 returned error can't find the container with id f20b2a1b26270b93cb023c1d9af702d7d1ebf34f65a9b8c156bded52a0c77fa0 Jul 16 21:52:09.893761 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:09.893731 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkshg" event={"ID":"7e676b3e-e7cf-41d1-ac8c-e2177080ea66","Type":"ContainerStarted","Data":"f20b2a1b26270b93cb023c1d9af702d7d1ebf34f65a9b8c156bded52a0c77fa0"} Jul 16 21:52:10.899010 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:10.898974 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkshg" event={"ID":"7e676b3e-e7cf-41d1-ac8c-e2177080ea66","Type":"ContainerStarted","Data":"eeca47779bb03b9bfd5a36bee332fb0eb43cd13b0676b253afbbed0d0745ee83"} Jul 16 21:52:10.899468 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:10.899043 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wkshg" event={"ID":"7e676b3e-e7cf-41d1-ac8c-e2177080ea66","Type":"ContainerStarted","Data":"251e662fc4e373181561e80588bf482fb0c4078441f3e5b3762d23578bbdc92d"} Jul 16 21:52:10.923220 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:10.923171 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wkshg" podStartSLOduration=129.913204648 podStartE2EDuration="2m10.923156984s" totalImagesPullingTime="1.009952336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:10.922043955 +0000 UTC m=+132.181954402" watchObservedRunningTime="2026-07-16 21:52:10.923156984 +0000 UTC m=+132.183067408" Jul 16 21:52:11.904927 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:11.904894 2558 generic.go:356] "Generic (PLEG): container finished" podID="5cd0ba43-b852-4efc-bfb9-0303473bb89c" containerID="25f641823085e2317f8d681d24cec5f5535fa4ce24dd7d0567d309d933a9c45c" exitCode=0 Jul 16 21:52:11.905365 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:11.904978 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" event={"ID":"5cd0ba43-b852-4efc-bfb9-0303473bb89c","Type":"ContainerDied","Data":"25f641823085e2317f8d681d24cec5f5535fa4ce24dd7d0567d309d933a9c45c"} Jul 16 21:52:11.905365 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:11.905336 2558 scope.go:117] "RemoveContainer" containerID="25f641823085e2317f8d681d24cec5f5535fa4ce24dd7d0567d309d933a9c45c" Jul 16 21:52:11.906714 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:11.906693 2558 generic.go:356] "Generic (PLEG): container finished" podID="37690e6c-a753-4329-8d1b-f327f3cf2c25" containerID="22d3b04a0b80bf70efbd1f43a1c1b8a86df3762511ef987e29f17e8063618257" exitCode=0 Jul 16 21:52:11.906794 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:11.906772 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" event={"ID":"37690e6c-a753-4329-8d1b-f327f3cf2c25","Type":"ContainerDied","Data":"22d3b04a0b80bf70efbd1f43a1c1b8a86df3762511ef987e29f17e8063618257"} Jul 16 21:52:11.907210 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:11.907197 2558 scope.go:117] "RemoveContainer" containerID="22d3b04a0b80bf70efbd1f43a1c1b8a86df3762511ef987e29f17e8063618257" Jul 16 21:52:12.913166 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:12.913136 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-7p689" event={"ID":"37690e6c-a753-4329-8d1b-f327f3cf2c25","Type":"ContainerStarted","Data":"a225a1968443568e84ccba7a26acf99c6e71a2034e4d80489fc579395ac48b75"} Jul 16 21:52:12.914907 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:12.914887 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-s9jq4" event={"ID":"5cd0ba43-b852-4efc-bfb9-0303473bb89c","Type":"ContainerStarted","Data":"2e0ce5e5b033122be7cff96efe16054c0063e2020202db878a727ce876bc5da8"} Jul 16 21:52:47.253267 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:47.253231 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:52:47.270486 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:47.270463 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:52:48.071986 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:52:48.071956 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:00.961017 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:00.960933 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:53:00.961493 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:00.961304 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="alertmanager" containerID="cri-o://f86fa0896876456d5fcc33fedef9e47ab26b1f1ab1a74d747faaca1c3e3c590c" gracePeriod=120 Jul 16 21:53:00.961493 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:00.961345 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy-web" containerID="cri-o://aa0cb5156c6b401fccd635b42460e082d4e1f876e1b5cee16a700342993c29dc" gracePeriod=120 Jul 16 21:53:00.961493 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:00.961346 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy-metric" containerID="cri-o://062a5221c771a8efda1ee896f3689de148dc885afe2248e61732df757e872f0f" gracePeriod=120 Jul 16 21:53:00.961493 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:00.961386 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy" containerID="cri-o://d1ed0587881083b7734ba99e11750fbcf1bcedce26407b07a6fe862704607b73" gracePeriod=120 Jul 16 21:53:00.961493 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:00.961419 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="config-reloader" containerID="cri-o://b4d7c55aa2e953ce4cfcc77334cad77632e8e6133c07582ecb3751bc5647360d" gracePeriod=120 Jul 16 21:53:00.961738 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:00.961511 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="prom-label-proxy" containerID="cri-o://1ed1a513d7b5eaf345c158ea7b468aa3bf959f07ec0a99c688c38907970985ba" gracePeriod=120 Jul 16 21:53:01.102757 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:01.102731 2558 generic.go:356] "Generic (PLEG): container finished" podID="97e58536-86af-4312-9a4c-5640b3fddd76" containerID="1ed1a513d7b5eaf345c158ea7b468aa3bf959f07ec0a99c688c38907970985ba" exitCode=0 Jul 16 21:53:01.102757 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:01.102754 2558 generic.go:356] "Generic (PLEG): container finished" podID="97e58536-86af-4312-9a4c-5640b3fddd76" containerID="d1ed0587881083b7734ba99e11750fbcf1bcedce26407b07a6fe862704607b73" exitCode=0 Jul 16 21:53:01.102757 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:01.102761 2558 generic.go:356] "Generic (PLEG): container finished" podID="97e58536-86af-4312-9a4c-5640b3fddd76" containerID="b4d7c55aa2e953ce4cfcc77334cad77632e8e6133c07582ecb3751bc5647360d" exitCode=0 Jul 16 21:53:01.102950 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:01.102768 2558 generic.go:356] "Generic (PLEG): container finished" podID="97e58536-86af-4312-9a4c-5640b3fddd76" containerID="f86fa0896876456d5fcc33fedef9e47ab26b1f1ab1a74d747faaca1c3e3c590c" exitCode=0 Jul 16 21:53:01.102950 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:01.102802 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerDied","Data":"1ed1a513d7b5eaf345c158ea7b468aa3bf959f07ec0a99c688c38907970985ba"} Jul 16 21:53:01.102950 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:01.102884 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerDied","Data":"d1ed0587881083b7734ba99e11750fbcf1bcedce26407b07a6fe862704607b73"} Jul 16 21:53:01.102950 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:01.102911 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerDied","Data":"b4d7c55aa2e953ce4cfcc77334cad77632e8e6133c07582ecb3751bc5647360d"} Jul 16 21:53:01.102950 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:01.102935 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerDied","Data":"f86fa0896876456d5fcc33fedef9e47ab26b1f1ab1a74d747faaca1c3e3c590c"} Jul 16 21:53:02.110340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.110305 2558 generic.go:356] "Generic (PLEG): container finished" podID="97e58536-86af-4312-9a4c-5640b3fddd76" containerID="062a5221c771a8efda1ee896f3689de148dc885afe2248e61732df757e872f0f" exitCode=0 Jul 16 21:53:02.110340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.110338 2558 generic.go:356] "Generic (PLEG): container finished" podID="97e58536-86af-4312-9a4c-5640b3fddd76" containerID="aa0cb5156c6b401fccd635b42460e082d4e1f876e1b5cee16a700342993c29dc" exitCode=0 Jul 16 21:53:02.110694 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.110371 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerDied","Data":"062a5221c771a8efda1ee896f3689de148dc885afe2248e61732df757e872f0f"} Jul 16 21:53:02.110694 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.110442 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerDied","Data":"aa0cb5156c6b401fccd635b42460e082d4e1f876e1b5cee16a700342993c29dc"} Jul 16 21:53:02.202537 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.202518 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:53:02.320130 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320050 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-main-db\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320130 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320083 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-trusted-ca-bundle\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320130 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320110 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-cluster-tls-config\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320144 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-web-config\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320173 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-web\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320190 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-config-out\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320247 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-config-volume\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320301 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-main-tls\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320343 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-metric\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320367 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwckm\" (UniqueName: \"kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-kube-api-access-fwckm\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320401 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320376 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:53:02.320988 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320410 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-tls-assets\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320988 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320455 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320988 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320452 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:02.320988 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320482 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-metrics-client-ca\") pod \"97e58536-86af-4312-9a4c-5640b3fddd76\" (UID: \"97e58536-86af-4312-9a4c-5640b3fddd76\") " Jul 16 21:53:02.320988 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320774 2558 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-main-db\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.320988 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.320793 2558 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.321299 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.321144 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:02.324093 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.324039 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:02.324093 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.324043 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-kube-api-access-fwckm" (OuterVolumeSpecName: "kube-api-access-fwckm") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "kube-api-access-fwckm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:53:02.324283 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.324080 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:02.324357 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.324302 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:02.324460 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.324439 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-config-volume" (OuterVolumeSpecName: "config-volume") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:02.324677 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.324652 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-config-out" (OuterVolumeSpecName: "config-out") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:53:02.324790 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.324749 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:53:02.325326 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.325308 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:02.328844 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.328697 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:02.334753 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.334734 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-web-config" (OuterVolumeSpecName: "web-config") pod "97e58536-86af-4312-9a4c-5640b3fddd76" (UID: "97e58536-86af-4312-9a4c-5640b3fddd76"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:02.421500 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.421467 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.421500 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.421496 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97e58536-86af-4312-9a4c-5640b3fddd76-metrics-client-ca\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.421500 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.421506 2558 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-cluster-tls-config\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.421716 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.421515 2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-web-config\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.421716 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.421525 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.421716 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.421534 2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/97e58536-86af-4312-9a4c-5640b3fddd76-config-out\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.421716 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.421542 2558 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-config-volume\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.421716 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.421552 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-main-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.421716 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.421560 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/97e58536-86af-4312-9a4c-5640b3fddd76-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.421716 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.421569 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwckm\" (UniqueName: \"kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-kube-api-access-fwckm\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:02.421716 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:02.421577 2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/97e58536-86af-4312-9a4c-5640b3fddd76-tls-assets\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:03.118355 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.118325 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"97e58536-86af-4312-9a4c-5640b3fddd76","Type":"ContainerDied","Data":"60e702ccc973ed16ba1833544be86abec6328fa269424a9923c4c523b40e03c9"} Jul 16 21:53:03.118355 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.118341 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 21:53:03.118800 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.118409 2558 scope.go:117] "RemoveContainer" containerID="1ed1a513d7b5eaf345c158ea7b468aa3bf959f07ec0a99c688c38907970985ba" Jul 16 21:53:03.126368 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.126340 2558 scope.go:117] "RemoveContainer" containerID="062a5221c771a8efda1ee896f3689de148dc885afe2248e61732df757e872f0f" Jul 16 21:53:03.134030 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.134009 2558 scope.go:117] "RemoveContainer" containerID="d1ed0587881083b7734ba99e11750fbcf1bcedce26407b07a6fe862704607b73" Jul 16 21:53:03.140496 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.140479 2558 scope.go:117] "RemoveContainer" containerID="aa0cb5156c6b401fccd635b42460e082d4e1f876e1b5cee16a700342993c29dc" Jul 16 21:53:03.145218 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.145186 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:53:03.148035 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.148020 2558 scope.go:117] "RemoveContainer" containerID="b4d7c55aa2e953ce4cfcc77334cad77632e8e6133c07582ecb3751bc5647360d" Jul 16 21:53:03.152427 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.152409 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 21:53:03.155706 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.155690 2558 scope.go:117] "RemoveContainer" containerID="f86fa0896876456d5fcc33fedef9e47ab26b1f1ab1a74d747faaca1c3e3c590c" Jul 16 21:53:03.162604 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.162588 2558 scope.go:117] "RemoveContainer" containerID="359246d55f7249d959ba340b907fa44c55afd95168b1a044893c455045b2991d" Jul 16 21:53:03.260635 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:03.260592 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" path="/var/lib/kubelet/pods/97e58536-86af-4312-9a4c-5640b3fddd76/volumes" Jul 16 21:53:05.351295 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.351261 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:53:05.351709 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.351528 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="prometheus" containerID="cri-o://b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03" gracePeriod=600 Jul 16 21:53:05.351709 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.351563 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="thanos-sidecar" containerID="cri-o://9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa" gracePeriod=600 Jul 16 21:53:05.351709 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.351596 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="config-reloader" containerID="cri-o://f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0" gracePeriod=600 Jul 16 21:53:05.351709 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.351630 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy-web" containerID="cri-o://659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237" gracePeriod=600 Jul 16 21:53:05.351709 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.351609 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d" gracePeriod=600 Jul 16 21:53:05.351709 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.351560 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy" containerID="cri-o://5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870" gracePeriod=600 Jul 16 21:53:05.592266 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.592245 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:05.750510 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750483 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-tls\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.750510 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750513 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-tls-assets\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.750745 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750536 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-trusted-ca-bundle\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.750745 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750552 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-web-config\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.750745 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750572 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-0\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.750745 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750597 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-kube-rbac-proxy\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.750745 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750616 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxw2n\" (UniqueName: \"kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-kube-api-access-hxw2n\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.750745 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750644 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-metrics-client-ca\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.750745 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750678 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.750745 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750710 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-db\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.750745 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750737 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-serving-certs-ca-bundle\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.751211 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750753 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-1\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.751211 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750792 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.751211 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750819 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-metrics-client-certs\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.751211 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750843 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-kubelet-serving-ca-bundle\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.751211 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750890 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-config\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.751211 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750921 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-config-out\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.751211 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750953 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-grpc-tls\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.751211 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.750990 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-thanos-prometheus-http-client-file\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.751211 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.751026 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-2\") pod \"c445dc85-473b-44dd-86fa-95df14dbf67f\" (UID: \"c445dc85-473b-44dd-86fa-95df14dbf67f\") " Jul 16 21:53:05.751612 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.751281 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:05.751713 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.751687 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:05.751825 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.751774 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:05.752629 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.752206 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:05.753385 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.753358 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:53:05.754030 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.754002 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:05.754245 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.754221 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:05.754317 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.754293 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:05.754378 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.754340 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:05.754810 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.754781 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:53:05.755374 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.755341 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:05.755374 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.755361 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-kube-api-access-hxw2n" (OuterVolumeSpecName: "kube-api-access-hxw2n") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "kube-api-access-hxw2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:53:05.755510 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.755360 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:05.755510 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.755497 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-config" (OuterVolumeSpecName: "config") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:05.755906 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.755881 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:05.755976 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.755951 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:53:05.756134 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.756116 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:05.756224 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.756205 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:05.756937 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.756918 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-config-out" (OuterVolumeSpecName: "config-out") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:53:05.764227 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.764207 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-web-config" (OuterVolumeSpecName: "web-config") pod "c445dc85-473b-44dd-86fa-95df14dbf67f" (UID: "c445dc85-473b-44dd-86fa-95df14dbf67f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:53:05.852073 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852042 2558 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852073 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852066 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852073 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852075 2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852085 2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-tls-assets\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852094 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852104 2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-web-config\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852112 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852120 2558 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-kube-rbac-proxy\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852128 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxw2n\" (UniqueName: \"kubernetes.io/projected/c445dc85-473b-44dd-86fa-95df14dbf67f-kube-api-access-hxw2n\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852136 2558 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-metrics-client-ca\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852145 2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852155 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-db\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852163 2558 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852171 2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852179 2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852188 2558 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-metrics-client-certs\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852196 2558 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c445dc85-473b-44dd-86fa-95df14dbf67f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852204 2558 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-config\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852211 2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c445dc85-473b-44dd-86fa-95df14dbf67f-config-out\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:05.852260 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:05.852219 2558 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c445dc85-473b-44dd-86fa-95df14dbf67f-secret-grpc-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 21:53:06.134051 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.133970 2558 generic.go:356] "Generic (PLEG): container finished" podID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerID="8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d" exitCode=0 Jul 16 21:53:06.134051 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.133998 2558 generic.go:356] "Generic (PLEG): container finished" podID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerID="5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870" exitCode=0 Jul 16 21:53:06.134051 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134008 2558 generic.go:356] "Generic (PLEG): container finished" podID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerID="659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237" exitCode=0 Jul 16 21:53:06.134051 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134017 2558 generic.go:356] "Generic (PLEG): container finished" podID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerID="9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa" exitCode=0 Jul 16 21:53:06.134051 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134025 2558 generic.go:356] "Generic (PLEG): container finished" podID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerID="f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0" exitCode=0 Jul 16 21:53:06.134051 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134033 2558 generic.go:356] "Generic (PLEG): container finished" podID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerID="b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03" exitCode=0 Jul 16 21:53:06.134340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134048 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerDied","Data":"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d"} Jul 16 21:53:06.134340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134063 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.134340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134120 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerDied","Data":"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870"} Jul 16 21:53:06.134340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134147 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerDied","Data":"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237"} Jul 16 21:53:06.134340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134177 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerDied","Data":"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa"} Jul 16 21:53:06.134340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134197 2558 scope.go:117] "RemoveContainer" containerID="8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d" Jul 16 21:53:06.134340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134214 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerDied","Data":"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0"} Jul 16 21:53:06.134340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134243 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerDied","Data":"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03"} Jul 16 21:53:06.134340 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.134267 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c445dc85-473b-44dd-86fa-95df14dbf67f","Type":"ContainerDied","Data":"95f9078f53961e6dbfebe30f4b9cbccbe3f9ade550d41420ecb18a729c6652b4"} Jul 16 21:53:06.142255 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.142232 2558 scope.go:117] "RemoveContainer" containerID="5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870" Jul 16 21:53:06.149166 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.149152 2558 scope.go:117] "RemoveContainer" containerID="659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237" Jul 16 21:53:06.155608 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.155590 2558 scope.go:117] "RemoveContainer" containerID="9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa" Jul 16 21:53:06.160412 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.160389 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:53:06.164497 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.164476 2558 scope.go:117] "RemoveContainer" containerID="f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0" Jul 16 21:53:06.169691 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.169662 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:53:06.175794 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.175775 2558 scope.go:117] "RemoveContainer" containerID="b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03" Jul 16 21:53:06.183384 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.183370 2558 scope.go:117] "RemoveContainer" containerID="19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a" Jul 16 21:53:06.190080 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.190066 2558 scope.go:117] "RemoveContainer" containerID="8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d" Jul 16 21:53:06.190350 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:53:06.190332 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": container with ID starting with 8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d not found: ID does not exist" containerID="8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d" Jul 16 21:53:06.190395 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.190359 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d"} err="failed to get container status \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": rpc error: code = NotFound desc = could not find container \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": container with ID starting with 8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d not found: ID does not exist" Jul 16 21:53:06.190395 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.190391 2558 scope.go:117] "RemoveContainer" containerID="5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870" Jul 16 21:53:06.190629 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:53:06.190609 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": container with ID starting with 5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870 not found: ID does not exist" containerID="5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870" Jul 16 21:53:06.190674 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.190636 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870"} err="failed to get container status \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": rpc error: code = NotFound desc = could not find container \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": container with ID starting with 5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870 not found: ID does not exist" Jul 16 21:53:06.190674 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.190653 2558 scope.go:117] "RemoveContainer" containerID="659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237" Jul 16 21:53:06.190893 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:53:06.190877 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": container with ID starting with 659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237 not found: ID does not exist" containerID="659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237" Jul 16 21:53:06.190939 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.190897 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237"} err="failed to get container status \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": rpc error: code = NotFound desc = could not find container \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": container with ID starting with 659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237 not found: ID does not exist" Jul 16 21:53:06.190939 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.190910 2558 scope.go:117] "RemoveContainer" containerID="9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa" Jul 16 21:53:06.191119 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:53:06.191101 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": container with ID starting with 9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa not found: ID does not exist" containerID="9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa" Jul 16 21:53:06.191203 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.191119 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa"} err="failed to get container status \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": rpc error: code = NotFound desc = could not find container \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": container with ID starting with 9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa not found: ID does not exist" Jul 16 21:53:06.191203 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.191131 2558 scope.go:117] "RemoveContainer" containerID="f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0" Jul 16 21:53:06.191368 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:53:06.191353 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": container with ID starting with f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0 not found: ID does not exist" containerID="f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0" Jul 16 21:53:06.191403 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.191372 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0"} err="failed to get container status \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": rpc error: code = NotFound desc = could not find container \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": container with ID starting with f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0 not found: ID does not exist" Jul 16 21:53:06.191403 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.191383 2558 scope.go:117] "RemoveContainer" containerID="b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03" Jul 16 21:53:06.191615 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:53:06.191598 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": container with ID starting with b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03 not found: ID does not exist" containerID="b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03" Jul 16 21:53:06.191668 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.191620 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03"} err="failed to get container status \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": rpc error: code = NotFound desc = could not find container \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": container with ID starting with b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03 not found: ID does not exist" Jul 16 21:53:06.191668 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.191635 2558 scope.go:117] "RemoveContainer" containerID="19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a" Jul 16 21:53:06.191843 ip-10-0-131-231 kubenswrapper[2558]: E0716 21:53:06.191822 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": container with ID starting with 19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a not found: ID does not exist" containerID="19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a" Jul 16 21:53:06.191899 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.191848 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a"} err="failed to get container status \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": rpc error: code = NotFound desc = could not find container \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": container with ID starting with 19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a not found: ID does not exist" Jul 16 21:53:06.191899 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.191884 2558 scope.go:117] "RemoveContainer" containerID="8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d" Jul 16 21:53:06.192092 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.192075 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d"} err="failed to get container status \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": rpc error: code = NotFound desc = could not find container \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": container with ID starting with 8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d not found: ID does not exist" Jul 16 21:53:06.192153 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.192094 2558 scope.go:117] "RemoveContainer" containerID="5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870" Jul 16 21:53:06.192324 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.192308 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870"} err="failed to get container status \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": rpc error: code = NotFound desc = could not find container \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": container with ID starting with 5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870 not found: ID does not exist" Jul 16 21:53:06.192397 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.192325 2558 scope.go:117] "RemoveContainer" containerID="659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237" Jul 16 21:53:06.192541 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.192526 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237"} err="failed to get container status \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": rpc error: code = NotFound desc = could not find container \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": container with ID starting with 659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237 not found: ID does not exist" Jul 16 21:53:06.192585 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.192541 2558 scope.go:117] "RemoveContainer" containerID="9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa" Jul 16 21:53:06.192749 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.192733 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa"} err="failed to get container status \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": rpc error: code = NotFound desc = could not find container \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": container with ID starting with 9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa not found: ID does not exist" Jul 16 21:53:06.192749 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.192748 2558 scope.go:117] "RemoveContainer" containerID="f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0" Jul 16 21:53:06.192953 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.192934 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0"} err="failed to get container status \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": rpc error: code = NotFound desc = could not find container \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": container with ID starting with f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0 not found: ID does not exist" Jul 16 21:53:06.192992 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.192955 2558 scope.go:117] "RemoveContainer" containerID="b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03" Jul 16 21:53:06.193186 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.193167 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03"} err="failed to get container status \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": rpc error: code = NotFound desc = could not find container \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": container with ID starting with b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03 not found: ID does not exist" Jul 16 21:53:06.193236 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.193187 2558 scope.go:117] "RemoveContainer" containerID="19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a" Jul 16 21:53:06.193400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.193384 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a"} err="failed to get container status \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": rpc error: code = NotFound desc = could not find container \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": container with ID starting with 19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a not found: ID does not exist" Jul 16 21:53:06.193400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.193399 2558 scope.go:117] "RemoveContainer" containerID="8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d" Jul 16 21:53:06.193600 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.193581 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d"} err="failed to get container status \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": rpc error: code = NotFound desc = could not find container \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": container with ID starting with 8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d not found: ID does not exist" Jul 16 21:53:06.193637 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.193601 2558 scope.go:117] "RemoveContainer" containerID="5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870" Jul 16 21:53:06.193791 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.193775 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870"} err="failed to get container status \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": rpc error: code = NotFound desc = could not find container \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": container with ID starting with 5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870 not found: ID does not exist" Jul 16 21:53:06.193834 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.193792 2558 scope.go:117] "RemoveContainer" containerID="659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237" Jul 16 21:53:06.193998 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.193982 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237"} err="failed to get container status \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": rpc error: code = NotFound desc = could not find container \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": container with ID starting with 659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237 not found: ID does not exist" Jul 16 21:53:06.194035 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.193998 2558 scope.go:117] "RemoveContainer" containerID="9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa" Jul 16 21:53:06.194204 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.194186 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa"} err="failed to get container status \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": rpc error: code = NotFound desc = could not find container \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": container with ID starting with 9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa not found: ID does not exist" Jul 16 21:53:06.194204 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.194203 2558 scope.go:117] "RemoveContainer" containerID="f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0" Jul 16 21:53:06.194420 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.194404 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0"} err="failed to get container status \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": rpc error: code = NotFound desc = could not find container \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": container with ID starting with f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0 not found: ID does not exist" Jul 16 21:53:06.194467 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.194420 2558 scope.go:117] "RemoveContainer" containerID="b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03" Jul 16 21:53:06.194628 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.194611 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03"} err="failed to get container status \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": rpc error: code = NotFound desc = could not find container \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": container with ID starting with b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03 not found: ID does not exist" Jul 16 21:53:06.194672 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.194629 2558 scope.go:117] "RemoveContainer" containerID="19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a" Jul 16 21:53:06.194831 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.194815 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a"} err="failed to get container status \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": rpc error: code = NotFound desc = could not find container \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": container with ID starting with 19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a not found: ID does not exist" Jul 16 21:53:06.194904 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.194830 2558 scope.go:117] "RemoveContainer" containerID="8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d" Jul 16 21:53:06.195052 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.195037 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d"} err="failed to get container status \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": rpc error: code = NotFound desc = could not find container \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": container with ID starting with 8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d not found: ID does not exist" Jul 16 21:53:06.195093 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.195052 2558 scope.go:117] "RemoveContainer" containerID="5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870" Jul 16 21:53:06.195253 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.195237 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870"} err="failed to get container status \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": rpc error: code = NotFound desc = could not find container \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": container with ID starting with 5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870 not found: ID does not exist" Jul 16 21:53:06.195308 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.195253 2558 scope.go:117] "RemoveContainer" containerID="659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237" Jul 16 21:53:06.195504 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.195471 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237"} err="failed to get container status \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": rpc error: code = NotFound desc = could not find container \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": container with ID starting with 659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237 not found: ID does not exist" Jul 16 21:53:06.195504 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.195503 2558 scope.go:117] "RemoveContainer" containerID="9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa" Jul 16 21:53:06.195757 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.195739 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa"} err="failed to get container status \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": rpc error: code = NotFound desc = could not find container \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": container with ID starting with 9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa not found: ID does not exist" Jul 16 21:53:06.195757 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.195755 2558 scope.go:117] "RemoveContainer" containerID="f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0" Jul 16 21:53:06.196119 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.196101 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0"} err="failed to get container status \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": rpc error: code = NotFound desc = could not find container \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": container with ID starting with f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0 not found: ID does not exist" Jul 16 21:53:06.196239 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.196121 2558 scope.go:117] "RemoveContainer" containerID="b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03" Jul 16 21:53:06.196336 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.196321 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03"} err="failed to get container status \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": rpc error: code = NotFound desc = could not find container \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": container with ID starting with b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03 not found: ID does not exist" Jul 16 21:53:06.196378 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.196337 2558 scope.go:117] "RemoveContainer" containerID="19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a" Jul 16 21:53:06.196525 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.196510 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a"} err="failed to get container status \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": rpc error: code = NotFound desc = could not find container \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": container with ID starting with 19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a not found: ID does not exist" Jul 16 21:53:06.196525 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.196523 2558 scope.go:117] "RemoveContainer" containerID="8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d" Jul 16 21:53:06.196698 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.196684 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d"} err="failed to get container status \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": rpc error: code = NotFound desc = could not find container \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": container with ID starting with 8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d not found: ID does not exist" Jul 16 21:53:06.196740 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.196697 2558 scope.go:117] "RemoveContainer" containerID="5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870" Jul 16 21:53:06.196936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.196912 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870"} err="failed to get container status \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": rpc error: code = NotFound desc = could not find container \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": container with ID starting with 5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870 not found: ID does not exist" Jul 16 21:53:06.196936 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.196936 2558 scope.go:117] "RemoveContainer" containerID="659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237" Jul 16 21:53:06.197141 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.197123 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237"} err="failed to get container status \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": rpc error: code = NotFound desc = could not find container \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": container with ID starting with 659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237 not found: ID does not exist" Jul 16 21:53:06.197141 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.197141 2558 scope.go:117] "RemoveContainer" containerID="9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa" Jul 16 21:53:06.197343 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.197322 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa"} err="failed to get container status \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": rpc error: code = NotFound desc = could not find container \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": container with ID starting with 9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa not found: ID does not exist" Jul 16 21:53:06.197416 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.197345 2558 scope.go:117] "RemoveContainer" containerID="f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0" Jul 16 21:53:06.197531 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.197515 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0"} err="failed to get container status \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": rpc error: code = NotFound desc = could not find container \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": container with ID starting with f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0 not found: ID does not exist" Jul 16 21:53:06.197590 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.197533 2558 scope.go:117] "RemoveContainer" containerID="b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03" Jul 16 21:53:06.197752 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.197737 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03"} err="failed to get container status \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": rpc error: code = NotFound desc = could not find container \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": container with ID starting with b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03 not found: ID does not exist" Jul 16 21:53:06.197815 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.197753 2558 scope.go:117] "RemoveContainer" containerID="19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a" Jul 16 21:53:06.197978 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.197961 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a"} err="failed to get container status \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": rpc error: code = NotFound desc = could not find container \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": container with ID starting with 19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a not found: ID does not exist" Jul 16 21:53:06.198019 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.197980 2558 scope.go:117] "RemoveContainer" containerID="8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d" Jul 16 21:53:06.198180 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.198163 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d"} err="failed to get container status \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": rpc error: code = NotFound desc = could not find container \"8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d\": container with ID starting with 8f3c0016a1b55efaa3573e7416492316ee31721862759d89bb2bcdbda2b7cd6d not found: ID does not exist" Jul 16 21:53:06.198245 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.198182 2558 scope.go:117] "RemoveContainer" containerID="5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870" Jul 16 21:53:06.198385 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.198370 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870"} err="failed to get container status \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": rpc error: code = NotFound desc = could not find container \"5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870\": container with ID starting with 5a9458b5d64b23866ba629bebc8229c1b1c87581cb9e2d2b7b0b91a9f5982870 not found: ID does not exist" Jul 16 21:53:06.198447 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.198387 2558 scope.go:117] "RemoveContainer" containerID="659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237" Jul 16 21:53:06.198592 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.198576 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237"} err="failed to get container status \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": rpc error: code = NotFound desc = could not find container \"659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237\": container with ID starting with 659b62526a55545e753a4a2b971e44b10d045d434d01bdae43f82ea0a05e2237 not found: ID does not exist" Jul 16 21:53:06.198638 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.198594 2558 scope.go:117] "RemoveContainer" containerID="9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa" Jul 16 21:53:06.198796 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.198779 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa"} err="failed to get container status \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": rpc error: code = NotFound desc = could not find container \"9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa\": container with ID starting with 9570f083183b6c2b47fb48a750ac03d1460b83331b7d9c338bfbf3a3e72574fa not found: ID does not exist" Jul 16 21:53:06.198885 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.198797 2558 scope.go:117] "RemoveContainer" containerID="f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0" Jul 16 21:53:06.199042 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.199027 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0"} err="failed to get container status \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": rpc error: code = NotFound desc = could not find container \"f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0\": container with ID starting with f2403b161401fe9ba3cc3b3998c4b83ba6a67540845579804e805ba8f4eb21c0 not found: ID does not exist" Jul 16 21:53:06.199082 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.199041 2558 scope.go:117] "RemoveContainer" containerID="b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03" Jul 16 21:53:06.199258 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.199237 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03"} err="failed to get container status \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": rpc error: code = NotFound desc = could not find container \"b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03\": container with ID starting with b0f83905fd850de64838ffa3da5d86f00364c1911c150267e36684252cd2bb03 not found: ID does not exist" Jul 16 21:53:06.199306 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.199260 2558 scope.go:117] "RemoveContainer" containerID="19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a" Jul 16 21:53:06.199452 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.199434 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a"} err="failed to get container status \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": rpc error: code = NotFound desc = could not find container \"19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a\": container with ID starting with 19773d9d0b3cc8c9fe4ba9f95286efe655a773ef0953169709e6d24f4b6fdd7a not found: ID does not exist" Jul 16 21:53:06.207609 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207590 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:53:06.207888 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207877 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="init-config-reloader" Jul 16 21:53:06.207938 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207890 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="init-config-reloader" Jul 16 21:53:06.207938 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207905 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy-web" Jul 16 21:53:06.207938 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207910 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy-web" Jul 16 21:53:06.207938 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207916 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="alertmanager" Jul 16 21:53:06.207938 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207921 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="alertmanager" Jul 16 21:53:06.207938 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207927 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="init-config-reloader" Jul 16 21:53:06.207938 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207932 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="init-config-reloader" Jul 16 21:53:06.207938 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207938 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207943 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207951 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy-metric" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207956 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy-metric" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207962 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy-thanos" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207967 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy-thanos" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207973 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207980 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207986 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="prom-label-proxy" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207991 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="prom-label-proxy" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.207997 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="prometheus" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208002 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="prometheus" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208007 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="thanos-sidecar" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208011 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="thanos-sidecar" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208018 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="config-reloader" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208023 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="config-reloader" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208028 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy-web" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208033 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy-web" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208038 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="config-reloader" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208043 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="config-reloader" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208080 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy-thanos" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208088 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="config-reloader" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208093 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="alertmanager" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208099 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208103 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="thanos-sidecar" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208110 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy-metric" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208115 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="prometheus" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208120 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="kube-rbac-proxy-web" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208124 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208130 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e58536-86af-4312-9a4c-5640b3fddd76" containerName="prom-label-proxy" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208134 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="config-reloader" Jul 16 21:53:06.208185 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.208138 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" containerName="kube-rbac-proxy-web" Jul 16 21:53:06.212157 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.212145 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.214750 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.214727 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 21:53:06.214892 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.214876 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 21:53:06.217722 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.217701 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 21:53:06.217827 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.217726 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 21:53:06.217827 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.217734 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 21:53:06.217827 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.217815 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 21:53:06.218256 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.218241 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f92rk\"" Jul 16 21:53:06.218319 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.218254 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 21:53:06.218367 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.218241 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 21:53:06.218541 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.218519 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 21:53:06.218823 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.218806 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-21nail2o0jl7a\"" Jul 16 21:53:06.219300 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.219258 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 21:53:06.219368 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.219271 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 21:53:06.219368 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.219363 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 21:53:06.219501 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.219486 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 21:53:06.221449 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.221431 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 21:53:06.223149 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.223135 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 21:53:06.250334 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.250303 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:53:06.355755 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.355723 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-web-config\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.355755 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.355756 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356170 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.355775 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356170 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.355844 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356170 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.355901 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356170 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.355998 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-config\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356170 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356021 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356170 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356089 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356170 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356134 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356170 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356165 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ktw\" (UniqueName: \"kubernetes.io/projected/aeebd685-e312-4848-adbb-2b74fccf2079-kube-api-access-l2ktw\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356206 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aeebd685-e312-4848-adbb-2b74fccf2079-config-out\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356228 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aeebd685-e312-4848-adbb-2b74fccf2079-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356244 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356276 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356297 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356356 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356400 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356398 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356618 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356416 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356618 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356440 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.356618 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.356500 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.457641 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.457618 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.457743 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.457645 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.457743 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.457664 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-config\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.457743 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.457686 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.457892 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.457802 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.457892 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.457840 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.457999 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.457890 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ktw\" (UniqueName: \"kubernetes.io/projected/aeebd685-e312-4848-adbb-2b74fccf2079-kube-api-access-l2ktw\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.457999 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.457923 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aeebd685-e312-4848-adbb-2b74fccf2079-config-out\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.457999 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.457948 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aeebd685-e312-4848-adbb-2b74fccf2079-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.457999 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.457971 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.458192 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.457997 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.458192 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.458023 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.458192 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.458068 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.458192 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.458108 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.458192 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.458135 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.458192 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.458162 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.458461 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.458196 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.458461 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.458230 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-web-config\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.458461 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.458257 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.458461 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.458283 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.458461 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.458427 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.459247 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.458929 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.459247 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.459139 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.459247 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.459199 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.460980 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.459618 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.460980 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.460052 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.460980 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.460755 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.461965 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.461925 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-config\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.462513 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.462490 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.463048 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.463026 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aeebd685-e312-4848-adbb-2b74fccf2079-config-out\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.463143 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.463122 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.463191 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.463171 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.463777 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.463602 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.463777 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.463631 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.463777 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.463729 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.463979 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.463911 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.464015 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.463978 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aeebd685-e312-4848-adbb-2b74fccf2079-web-config\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.464111 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.464095 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/aeebd685-e312-4848-adbb-2b74fccf2079-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.464713 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.464695 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aeebd685-e312-4848-adbb-2b74fccf2079-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.468890 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.468831 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ktw\" (UniqueName: \"kubernetes.io/projected/aeebd685-e312-4848-adbb-2b74fccf2079-kube-api-access-l2ktw\") pod \"prometheus-k8s-0\" (UID: \"aeebd685-e312-4848-adbb-2b74fccf2079\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.521735 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.521715 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:53:06.663330 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:06.663307 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 21:53:06.665742 ip-10-0-131-231 kubenswrapper[2558]: W0716 21:53:06.665719 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeebd685_e312_4848_adbb_2b74fccf2079.slice/crio-f9074f79e3c6dd86e51dec5d70daf21fa28add32b7e54c363d301a6dd6effd96 WatchSource:0}: Error finding container f9074f79e3c6dd86e51dec5d70daf21fa28add32b7e54c363d301a6dd6effd96: Status 404 returned error can't find the container with id f9074f79e3c6dd86e51dec5d70daf21fa28add32b7e54c363d301a6dd6effd96 Jul 16 21:53:07.140529 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:07.140442 2558 generic.go:356] "Generic (PLEG): container finished" podID="aeebd685-e312-4848-adbb-2b74fccf2079" containerID="76bb0647d93985c19fc966bde2e3d22adbb759043e1c1dcfa09ad8036dc3e55c" exitCode=0 Jul 16 21:53:07.140529 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:07.140496 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aeebd685-e312-4848-adbb-2b74fccf2079","Type":"ContainerDied","Data":"76bb0647d93985c19fc966bde2e3d22adbb759043e1c1dcfa09ad8036dc3e55c"} Jul 16 21:53:07.140738 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:07.140543 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aeebd685-e312-4848-adbb-2b74fccf2079","Type":"ContainerStarted","Data":"f9074f79e3c6dd86e51dec5d70daf21fa28add32b7e54c363d301a6dd6effd96"} Jul 16 21:53:07.269740 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:07.269703 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c445dc85-473b-44dd-86fa-95df14dbf67f" path="/var/lib/kubelet/pods/c445dc85-473b-44dd-86fa-95df14dbf67f/volumes" Jul 16 21:53:08.147613 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:08.147578 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aeebd685-e312-4848-adbb-2b74fccf2079","Type":"ContainerStarted","Data":"9ec6b3e44f749e8a905280883d2efe5dec6c3c0108decf77bdd7c548a630bb0d"} Jul 16 21:53:08.148066 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:08.147637 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aeebd685-e312-4848-adbb-2b74fccf2079","Type":"ContainerStarted","Data":"f67594dbbc367f74f27247aa1b5cab038aa9b19a6e596669f9f6a88b9f3f7cb6"} Jul 16 21:53:08.148066 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:08.147664 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aeebd685-e312-4848-adbb-2b74fccf2079","Type":"ContainerStarted","Data":"a546e0eea55e6ac4d623e076443cdd0207f3730191c1c11a71511091f0efb665"} Jul 16 21:53:08.148066 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:08.147694 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aeebd685-e312-4848-adbb-2b74fccf2079","Type":"ContainerStarted","Data":"56205ea56faa1df91d591f243906085aa7318664abebeeac220e13d855368c27"} Jul 16 21:53:08.148066 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:08.147719 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aeebd685-e312-4848-adbb-2b74fccf2079","Type":"ContainerStarted","Data":"12e20208b48b46e9c00a962768d7977b47f09d45b651056b78648c161e2b012d"} Jul 16 21:53:08.148066 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:08.147742 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"aeebd685-e312-4848-adbb-2b74fccf2079","Type":"ContainerStarted","Data":"259b453c2c124d45057596ca7857e51961137fb4c7a1e8995a84b724277eff19"} Jul 16 21:53:08.179091 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:08.179015 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.159263525 podStartE2EDuration="2.178999894s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.736369ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:08.177152815 +0000 UTC m=+189.437063239" watchObservedRunningTime="2026-07-16 21:53:08.178999894 +0000 UTC m=+189.438910318" Jul 16 21:53:11.522019 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:53:11.521972 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:54:06.522961 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:54:06.522917 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:54:06.538596 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:54:06.538574 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:54:07.386111 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:54:07.386085 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 21:54:59.099894 ip-10-0-131-231 kubenswrapper[2558]: I0716 21:54:59.099851 2558 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 22:01:36.862053 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:36.862011 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd"] Jul 16 22:01:36.865209 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:36.865193 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:01:36.871342 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:36.871316 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d9a7f-kube-rbac-proxy-sar-config\"" Jul 16 22:01:36.871478 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:36.871374 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 16 22:01:36.872111 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:36.872081 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d9a7f-serving-cert\"" Jul 16 22:01:36.872111 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:36.872092 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-65f4r\"" Jul 16 22:01:36.878023 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:36.877803 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd"] Jul 16 22:01:36.916945 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:36.916914 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2767c720-5514-4c08-8943-641933c19a7e-proxy-tls\") pod \"switch-graph-d9a7f-56f7cf8887-dnrnd\" (UID: \"2767c720-5514-4c08-8943-641933c19a7e\") " pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:01:36.917070 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:36.916951 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2767c720-5514-4c08-8943-641933c19a7e-openshift-service-ca-bundle\") pod \"switch-graph-d9a7f-56f7cf8887-dnrnd\" (UID: \"2767c720-5514-4c08-8943-641933c19a7e\") " pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:01:37.018146 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:37.018108 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2767c720-5514-4c08-8943-641933c19a7e-proxy-tls\") pod \"switch-graph-d9a7f-56f7cf8887-dnrnd\" (UID: \"2767c720-5514-4c08-8943-641933c19a7e\") " pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:01:37.018146 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:37.018148 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2767c720-5514-4c08-8943-641933c19a7e-openshift-service-ca-bundle\") pod \"switch-graph-d9a7f-56f7cf8887-dnrnd\" (UID: \"2767c720-5514-4c08-8943-641933c19a7e\") " pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:01:37.018355 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:01:37.018269 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-d9a7f-serving-cert: secret "switch-graph-d9a7f-serving-cert" not found Jul 16 22:01:37.018355 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:01:37.018352 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2767c720-5514-4c08-8943-641933c19a7e-proxy-tls podName:2767c720-5514-4c08-8943-641933c19a7e nodeName:}" failed. No retries permitted until 2026-07-16 22:01:37.518331949 +0000 UTC m=+698.778242354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2767c720-5514-4c08-8943-641933c19a7e-proxy-tls") pod "switch-graph-d9a7f-56f7cf8887-dnrnd" (UID: "2767c720-5514-4c08-8943-641933c19a7e") : secret "switch-graph-d9a7f-serving-cert" not found Jul 16 22:01:37.018888 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:37.018872 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2767c720-5514-4c08-8943-641933c19a7e-openshift-service-ca-bundle\") pod \"switch-graph-d9a7f-56f7cf8887-dnrnd\" (UID: \"2767c720-5514-4c08-8943-641933c19a7e\") " pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:01:37.522820 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:37.522786 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2767c720-5514-4c08-8943-641933c19a7e-proxy-tls\") pod \"switch-graph-d9a7f-56f7cf8887-dnrnd\" (UID: \"2767c720-5514-4c08-8943-641933c19a7e\") " pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:01:37.525327 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:37.525295 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2767c720-5514-4c08-8943-641933c19a7e-proxy-tls\") pod \"switch-graph-d9a7f-56f7cf8887-dnrnd\" (UID: \"2767c720-5514-4c08-8943-641933c19a7e\") " pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:01:37.782100 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:37.782003 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:01:37.914104 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:37.914075 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd"] Jul 16 22:01:37.917274 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:01:37.917249 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2767c720_5514_4c08_8943_641933c19a7e.slice/crio-9dbc75b91ac5fa72f90e08cea2f08e1f0d3ea2d3beb7d7a4915e74d30ffea6b2 WatchSource:0}: Error finding container 9dbc75b91ac5fa72f90e08cea2f08e1f0d3ea2d3beb7d7a4915e74d30ffea6b2: Status 404 returned error can't find the container with id 9dbc75b91ac5fa72f90e08cea2f08e1f0d3ea2d3beb7d7a4915e74d30ffea6b2 Jul 16 22:01:37.919143 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:37.919128 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 22:01:37.998996 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:37.998958 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" event={"ID":"2767c720-5514-4c08-8943-641933c19a7e","Type":"ContainerStarted","Data":"9dbc75b91ac5fa72f90e08cea2f08e1f0d3ea2d3beb7d7a4915e74d30ffea6b2"} Jul 16 22:01:41.013541 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:41.013509 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" event={"ID":"2767c720-5514-4c08-8943-641933c19a7e","Type":"ContainerStarted","Data":"fc0966149eddf2c3492ab71c1de9ccd8f19b144bf9db872d8431a24ec68a7954"} Jul 16 22:01:41.014007 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:41.013634 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:01:41.030779 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:41.030731 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" podStartSLOduration=2.551989281 podStartE2EDuration="5.03071816s" totalImagesPullingTime="2.478728879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:01:41.02945466 +0000 UTC m=+702.289365084" watchObservedRunningTime="2026-07-16 22:01:41.03071816 +0000 UTC m=+702.290628584" Jul 16 22:01:47.023252 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:47.023227 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:01:51.094321 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:51.094239 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd"] Jul 16 22:01:51.094745 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:51.094501 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" podUID="2767c720-5514-4c08-8943-641933c19a7e" containerName="switch-graph-d9a7f" containerID="cri-o://fc0966149eddf2c3492ab71c1de9ccd8f19b144bf9db872d8431a24ec68a7954" gracePeriod=30 Jul 16 22:01:52.021092 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:52.021056 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" podUID="2767c720-5514-4c08-8943-641933c19a7e" containerName="switch-graph-d9a7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:01:57.021102 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:01:57.021069 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" podUID="2767c720-5514-4c08-8943-641933c19a7e" containerName="switch-graph-d9a7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:02:02.021333 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:02.021293 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" podUID="2767c720-5514-4c08-8943-641933c19a7e" containerName="switch-graph-d9a7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:02:02.021732 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:02.021396 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:02:07.021096 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:07.021060 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" podUID="2767c720-5514-4c08-8943-641933c19a7e" containerName="switch-graph-d9a7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:02:12.021551 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:12.021514 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" podUID="2767c720-5514-4c08-8943-641933c19a7e" containerName="switch-graph-d9a7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:02:17.021586 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:17.021550 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" podUID="2767c720-5514-4c08-8943-641933c19a7e" containerName="switch-graph-d9a7f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:02:21.159971 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:21.159944 2558 generic.go:356] "Generic (PLEG): container finished" podID="2767c720-5514-4c08-8943-641933c19a7e" containerID="fc0966149eddf2c3492ab71c1de9ccd8f19b144bf9db872d8431a24ec68a7954" exitCode=0 Jul 16 22:02:21.160280 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:21.160013 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" event={"ID":"2767c720-5514-4c08-8943-641933c19a7e","Type":"ContainerDied","Data":"fc0966149eddf2c3492ab71c1de9ccd8f19b144bf9db872d8431a24ec68a7954"} Jul 16 22:02:21.285044 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:21.285016 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:02:21.376133 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:21.376064 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2767c720-5514-4c08-8943-641933c19a7e-proxy-tls\") pod \"2767c720-5514-4c08-8943-641933c19a7e\" (UID: \"2767c720-5514-4c08-8943-641933c19a7e\") " Jul 16 22:02:21.376133 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:21.376103 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2767c720-5514-4c08-8943-641933c19a7e-openshift-service-ca-bundle\") pod \"2767c720-5514-4c08-8943-641933c19a7e\" (UID: \"2767c720-5514-4c08-8943-641933c19a7e\") " Jul 16 22:02:21.376549 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:21.376518 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2767c720-5514-4c08-8943-641933c19a7e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2767c720-5514-4c08-8943-641933c19a7e" (UID: "2767c720-5514-4c08-8943-641933c19a7e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:02:21.376677 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:21.376595 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2767c720-5514-4c08-8943-641933c19a7e-openshift-service-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:02:21.378417 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:21.378387 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2767c720-5514-4c08-8943-641933c19a7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2767c720-5514-4c08-8943-641933c19a7e" (UID: "2767c720-5514-4c08-8943-641933c19a7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:02:21.477026 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:21.476997 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2767c720-5514-4c08-8943-641933c19a7e-proxy-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:02:22.165681 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:22.165644 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" event={"ID":"2767c720-5514-4c08-8943-641933c19a7e","Type":"ContainerDied","Data":"9dbc75b91ac5fa72f90e08cea2f08e1f0d3ea2d3beb7d7a4915e74d30ffea6b2"} Jul 16 22:02:22.166115 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:22.165731 2558 scope.go:117] "RemoveContainer" containerID="fc0966149eddf2c3492ab71c1de9ccd8f19b144bf9db872d8431a24ec68a7954" Jul 16 22:02:22.166115 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:22.165655 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd" Jul 16 22:02:22.186570 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:22.186548 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd"] Jul 16 22:02:22.190154 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:22.190133 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d9a7f-56f7cf8887-dnrnd"] Jul 16 22:02:23.260317 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:23.260282 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2767c720-5514-4c08-8943-641933c19a7e" path="/var/lib/kubelet/pods/2767c720-5514-4c08-8943-641933c19a7e/volumes" Jul 16 22:02:26.866746 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:26.866710 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk"] Jul 16 22:02:26.867204 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:26.867188 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2767c720-5514-4c08-8943-641933c19a7e" containerName="switch-graph-d9a7f" Jul 16 22:02:26.867249 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:26.867210 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2767c720-5514-4c08-8943-641933c19a7e" containerName="switch-graph-d9a7f" Jul 16 22:02:26.867292 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:26.867279 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2767c720-5514-4c08-8943-641933c19a7e" containerName="switch-graph-d9a7f" Jul 16 22:02:26.869504 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:26.869487 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:02:26.871620 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:26.871599 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 16 22:02:26.871620 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:26.871609 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Jul 16 22:02:26.871789 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:26.871650 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Jul 16 22:02:26.871951 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:26.871935 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-65f4r\"" Jul 16 22:02:26.882251 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:26.882225 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk"] Jul 16 22:02:27.021687 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:27.021653 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-openshift-service-ca-bundle\") pod \"model-chainer-5b578b95c5-5vjwk\" (UID: \"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b\") " pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:02:27.021846 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:27.021766 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-proxy-tls\") pod \"model-chainer-5b578b95c5-5vjwk\" (UID: \"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b\") " pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:02:27.122242 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:27.122155 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-proxy-tls\") pod \"model-chainer-5b578b95c5-5vjwk\" (UID: \"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b\") " pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:02:27.122242 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:27.122210 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-openshift-service-ca-bundle\") pod \"model-chainer-5b578b95c5-5vjwk\" (UID: \"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b\") " pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:02:27.122848 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:27.122823 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-openshift-service-ca-bundle\") pod \"model-chainer-5b578b95c5-5vjwk\" (UID: \"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b\") " pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:02:27.124646 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:27.124615 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-proxy-tls\") pod \"model-chainer-5b578b95c5-5vjwk\" (UID: \"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b\") " pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:02:27.186965 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:27.186935 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:02:27.316360 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:27.316327 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk"] Jul 16 22:02:27.319549 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:02:27.319520 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea7e54b_fa93_44fa_86fd_e3bd5c447a2b.slice/crio-a08dffa6a1a067490288dd16f92a3960ca0872c341ed492e6ab469d06bc25132 WatchSource:0}: Error finding container a08dffa6a1a067490288dd16f92a3960ca0872c341ed492e6ab469d06bc25132: Status 404 returned error can't find the container with id a08dffa6a1a067490288dd16f92a3960ca0872c341ed492e6ab469d06bc25132 Jul 16 22:02:28.189922 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:28.189875 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" event={"ID":"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b","Type":"ContainerStarted","Data":"77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05"} Jul 16 22:02:28.190343 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:28.189932 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" event={"ID":"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b","Type":"ContainerStarted","Data":"a08dffa6a1a067490288dd16f92a3960ca0872c341ed492e6ab469d06bc25132"} Jul 16 22:02:28.190343 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:28.189952 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:02:28.206131 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:28.206078 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" podStartSLOduration=2.206058505 podStartE2EDuration="2.206058505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:02:28.205777293 +0000 UTC m=+749.465687716" watchObservedRunningTime="2026-07-16 22:02:28.206058505 +0000 UTC m=+749.465968945" Jul 16 22:02:34.199776 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:34.199750 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:02:36.974239 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:36.974209 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk"] Jul 16 22:02:36.974601 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:36.974380 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" podUID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" containerName="model-chainer" containerID="cri-o://77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05" gracePeriod=30 Jul 16 22:02:39.198067 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:39.198025 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" podUID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:02:44.199016 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:44.198972 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" podUID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:02:49.198573 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:49.198537 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" podUID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:02:49.198996 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:49.198633 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:02:54.197657 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:54.197620 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" podUID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:02:59.198510 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:02:59.198480 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" podUID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:03:01.383069 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:01.383037 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn"] Jul 16 22:03:01.385875 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:01.385844 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:03:01.388198 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:01.388174 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-69517-serving-cert\"" Jul 16 22:03:01.388808 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:01.388787 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-69517-kube-rbac-proxy-sar-config\"" Jul 16 22:03:01.398520 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:01.398499 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn"] Jul 16 22:03:01.479332 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:01.479305 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ae74149-4ea3-4866-82a2-3682c63e516c-openshift-service-ca-bundle\") pod \"switch-graph-69517-c7754998c-6lqsn\" (UID: \"9ae74149-4ea3-4866-82a2-3682c63e516c\") " pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:03:01.479484 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:01.479353 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae74149-4ea3-4866-82a2-3682c63e516c-proxy-tls\") pod \"switch-graph-69517-c7754998c-6lqsn\" (UID: \"9ae74149-4ea3-4866-82a2-3682c63e516c\") " pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:03:01.580597 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:01.580561 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ae74149-4ea3-4866-82a2-3682c63e516c-openshift-service-ca-bundle\") pod \"switch-graph-69517-c7754998c-6lqsn\" (UID: \"9ae74149-4ea3-4866-82a2-3682c63e516c\") " pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:03:01.580754 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:01.580630 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae74149-4ea3-4866-82a2-3682c63e516c-proxy-tls\") pod \"switch-graph-69517-c7754998c-6lqsn\" (UID: \"9ae74149-4ea3-4866-82a2-3682c63e516c\") " pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:03:01.580801 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:03:01.580772 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-69517-serving-cert: secret "switch-graph-69517-serving-cert" not found Jul 16 22:03:01.580874 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:03:01.580843 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae74149-4ea3-4866-82a2-3682c63e516c-proxy-tls podName:9ae74149-4ea3-4866-82a2-3682c63e516c nodeName:}" failed. No retries permitted until 2026-07-16 22:03:02.080828121 +0000 UTC m=+783.340738524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9ae74149-4ea3-4866-82a2-3682c63e516c-proxy-tls") pod "switch-graph-69517-c7754998c-6lqsn" (UID: "9ae74149-4ea3-4866-82a2-3682c63e516c") : secret "switch-graph-69517-serving-cert" not found Jul 16 22:03:01.581191 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:01.581145 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ae74149-4ea3-4866-82a2-3682c63e516c-openshift-service-ca-bundle\") pod \"switch-graph-69517-c7754998c-6lqsn\" (UID: \"9ae74149-4ea3-4866-82a2-3682c63e516c\") " pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:03:02.084215 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:02.084181 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae74149-4ea3-4866-82a2-3682c63e516c-proxy-tls\") pod \"switch-graph-69517-c7754998c-6lqsn\" (UID: \"9ae74149-4ea3-4866-82a2-3682c63e516c\") " pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:03:02.086681 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:02.086660 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae74149-4ea3-4866-82a2-3682c63e516c-proxy-tls\") pod \"switch-graph-69517-c7754998c-6lqsn\" (UID: \"9ae74149-4ea3-4866-82a2-3682c63e516c\") " pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:03:02.295125 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:02.295096 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:03:02.422974 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:02.422945 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn"] Jul 16 22:03:02.428969 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:03:02.428933 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae74149_4ea3_4866_82a2_3682c63e516c.slice/crio-d1d46e16d9765b11e24d6a706a161c1987fd146b3e01d104d9ada8e9e05176be WatchSource:0}: Error finding container d1d46e16d9765b11e24d6a706a161c1987fd146b3e01d104d9ada8e9e05176be: Status 404 returned error can't find the container with id d1d46e16d9765b11e24d6a706a161c1987fd146b3e01d104d9ada8e9e05176be Jul 16 22:03:03.318390 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:03.318360 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" event={"ID":"9ae74149-4ea3-4866-82a2-3682c63e516c","Type":"ContainerStarted","Data":"3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367"} Jul 16 22:03:03.318563 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:03.318407 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" event={"ID":"9ae74149-4ea3-4866-82a2-3682c63e516c","Type":"ContainerStarted","Data":"d1d46e16d9765b11e24d6a706a161c1987fd146b3e01d104d9ada8e9e05176be"} Jul 16 22:03:03.318563 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:03.318497 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:03:03.336523 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:03.336475 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" podStartSLOduration=2.336460193 podStartE2EDuration="2.336460193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:03:03.334950426 +0000 UTC m=+784.594860875" watchObservedRunningTime="2026-07-16 22:03:03.336460193 +0000 UTC m=+784.596370618" Jul 16 22:03:04.198079 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:04.198039 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" podUID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:03:07.159088 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.159059 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:03:07.226611 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.226568 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-openshift-service-ca-bundle\") pod \"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b\" (UID: \"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b\") " Jul 16 22:03:07.226792 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.226661 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-proxy-tls\") pod \"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b\" (UID: \"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b\") " Jul 16 22:03:07.226970 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.226942 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" (UID: "1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:03:07.229116 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.229067 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" (UID: "1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:03:07.327210 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.327177 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-openshift-service-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:03:07.327210 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.327207 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b-proxy-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:03:07.333264 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.333241 2558 generic.go:356] "Generic (PLEG): container finished" podID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" containerID="77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05" exitCode=137 Jul 16 22:03:07.333381 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.333286 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" Jul 16 22:03:07.333381 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.333319 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" event={"ID":"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b","Type":"ContainerDied","Data":"77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05"} Jul 16 22:03:07.333381 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.333372 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk" event={"ID":"1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b","Type":"ContainerDied","Data":"a08dffa6a1a067490288dd16f92a3960ca0872c341ed492e6ab469d06bc25132"} Jul 16 22:03:07.333497 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.333393 2558 scope.go:117] "RemoveContainer" containerID="77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05" Jul 16 22:03:07.346186 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.346158 2558 scope.go:117] "RemoveContainer" containerID="77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05" Jul 16 22:03:07.346593 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:03:07.346561 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05\": container with ID starting with 77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05 not found: ID does not exist" containerID="77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05" Jul 16 22:03:07.346673 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.346609 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05"} err="failed to get container status \"77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05\": rpc error: code = NotFound desc = could not find container \"77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05\": container with ID starting with 77136b04405f40d1fb9af2391518f21801b453a4c5d4fe4ade70c7900ea9bf05 not found: ID does not exist" Jul 16 22:03:07.351485 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.351461 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk"] Jul 16 22:03:07.353731 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:07.353713 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-5b578b95c5-5vjwk"] Jul 16 22:03:09.260732 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:09.260695 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" path="/var/lib/kubelet/pods/1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b/volumes" Jul 16 22:03:09.327797 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:09.327771 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:03:37.211868 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.211777 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82"] Jul 16 22:03:37.212291 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.212105 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" containerName="model-chainer" Jul 16 22:03:37.212291 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.212116 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" containerName="model-chainer" Jul 16 22:03:37.212291 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.212164 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ea7e54b-fa93-44fa-86fd-e3bd5c447a2b" containerName="model-chainer" Jul 16 22:03:37.213930 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.213916 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:03:37.216290 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.216265 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-77f35-kube-rbac-proxy-sar-config\"" Jul 16 22:03:37.216404 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.216305 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-77f35-serving-cert\"" Jul 16 22:03:37.235380 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.235353 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82"] Jul 16 22:03:37.375730 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.375700 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da114ca6-b44d-4a4b-8415-3e17a65a6995-openshift-service-ca-bundle\") pod \"sequence-graph-77f35-5678976b5d-lxn82\" (UID: \"da114ca6-b44d-4a4b-8415-3e17a65a6995\") " pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:03:37.375929 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.375751 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da114ca6-b44d-4a4b-8415-3e17a65a6995-proxy-tls\") pod \"sequence-graph-77f35-5678976b5d-lxn82\" (UID: \"da114ca6-b44d-4a4b-8415-3e17a65a6995\") " pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:03:37.476234 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.476147 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da114ca6-b44d-4a4b-8415-3e17a65a6995-openshift-service-ca-bundle\") pod \"sequence-graph-77f35-5678976b5d-lxn82\" (UID: \"da114ca6-b44d-4a4b-8415-3e17a65a6995\") " pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:03:37.476234 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.476206 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da114ca6-b44d-4a4b-8415-3e17a65a6995-proxy-tls\") pod \"sequence-graph-77f35-5678976b5d-lxn82\" (UID: \"da114ca6-b44d-4a4b-8415-3e17a65a6995\") " pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:03:37.476849 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.476824 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da114ca6-b44d-4a4b-8415-3e17a65a6995-openshift-service-ca-bundle\") pod \"sequence-graph-77f35-5678976b5d-lxn82\" (UID: \"da114ca6-b44d-4a4b-8415-3e17a65a6995\") " pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:03:37.479059 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.479037 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da114ca6-b44d-4a4b-8415-3e17a65a6995-proxy-tls\") pod \"sequence-graph-77f35-5678976b5d-lxn82\" (UID: \"da114ca6-b44d-4a4b-8415-3e17a65a6995\") " pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:03:37.523155 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.523124 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:03:37.650118 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:37.650056 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82"] Jul 16 22:03:37.653718 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:03:37.653688 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda114ca6_b44d_4a4b_8415_3e17a65a6995.slice/crio-253c8fd4145bdbc8db2316cb4a8d25db6004df9915f2ee7590e477a836d1601b WatchSource:0}: Error finding container 253c8fd4145bdbc8db2316cb4a8d25db6004df9915f2ee7590e477a836d1601b: Status 404 returned error can't find the container with id 253c8fd4145bdbc8db2316cb4a8d25db6004df9915f2ee7590e477a836d1601b Jul 16 22:03:38.449150 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:38.449111 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" event={"ID":"da114ca6-b44d-4a4b-8415-3e17a65a6995","Type":"ContainerStarted","Data":"39f955cc92fab75c25be6bbf470d318ee9d272160ae264d9b2856d2946f2de50"} Jul 16 22:03:38.449549 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:38.449163 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" event={"ID":"da114ca6-b44d-4a4b-8415-3e17a65a6995","Type":"ContainerStarted","Data":"253c8fd4145bdbc8db2316cb4a8d25db6004df9915f2ee7590e477a836d1601b"} Jul 16 22:03:38.449549 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:38.449235 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:03:38.466433 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:38.466377 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" podStartSLOduration=1.4663596810000001 podStartE2EDuration="1.466359681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:03:38.465759948 +0000 UTC m=+819.725670371" watchObservedRunningTime="2026-07-16 22:03:38.466359681 +0000 UTC m=+819.726270100" Jul 16 22:03:44.458428 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:03:44.458396 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:11:15.939736 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:15.939689 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn"] Jul 16 22:11:15.940340 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:15.940021 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" podUID="9ae74149-4ea3-4866-82a2-3682c63e516c" containerName="switch-graph-69517" containerID="cri-o://3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367" gracePeriod=30 Jul 16 22:11:19.325110 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:19.325073 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" podUID="9ae74149-4ea3-4866-82a2-3682c63e516c" containerName="switch-graph-69517" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:11:24.325744 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:24.325703 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" podUID="9ae74149-4ea3-4866-82a2-3682c63e516c" containerName="switch-graph-69517" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:11:29.325734 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:29.325700 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" podUID="9ae74149-4ea3-4866-82a2-3682c63e516c" containerName="switch-graph-69517" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:11:29.326194 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:29.325804 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:11:34.325085 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:34.325048 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" podUID="9ae74149-4ea3-4866-82a2-3682c63e516c" containerName="switch-graph-69517" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:11:39.325733 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:39.325699 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" podUID="9ae74149-4ea3-4866-82a2-3682c63e516c" containerName="switch-graph-69517" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:11:44.325909 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:44.325845 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" podUID="9ae74149-4ea3-4866-82a2-3682c63e516c" containerName="switch-graph-69517" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:11:46.087391 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.087370 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:11:46.173178 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.173157 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ae74149-4ea3-4866-82a2-3682c63e516c-openshift-service-ca-bundle\") pod \"9ae74149-4ea3-4866-82a2-3682c63e516c\" (UID: \"9ae74149-4ea3-4866-82a2-3682c63e516c\") " Jul 16 22:11:46.173341 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.173250 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae74149-4ea3-4866-82a2-3682c63e516c-proxy-tls\") pod \"9ae74149-4ea3-4866-82a2-3682c63e516c\" (UID: \"9ae74149-4ea3-4866-82a2-3682c63e516c\") " Jul 16 22:11:46.173498 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.173475 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ae74149-4ea3-4866-82a2-3682c63e516c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9ae74149-4ea3-4866-82a2-3682c63e516c" (UID: "9ae74149-4ea3-4866-82a2-3682c63e516c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:11:46.175391 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.175367 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ae74149-4ea3-4866-82a2-3682c63e516c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ae74149-4ea3-4866-82a2-3682c63e516c" (UID: "9ae74149-4ea3-4866-82a2-3682c63e516c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:11:46.225504 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.225449 2558 generic.go:356] "Generic (PLEG): container finished" podID="9ae74149-4ea3-4866-82a2-3682c63e516c" containerID="3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367" exitCode=0 Jul 16 22:11:46.225504 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.225491 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" Jul 16 22:11:46.225504 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.225499 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" event={"ID":"9ae74149-4ea3-4866-82a2-3682c63e516c","Type":"ContainerDied","Data":"3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367"} Jul 16 22:11:46.225666 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.225541 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn" event={"ID":"9ae74149-4ea3-4866-82a2-3682c63e516c","Type":"ContainerDied","Data":"d1d46e16d9765b11e24d6a706a161c1987fd146b3e01d104d9ada8e9e05176be"} Jul 16 22:11:46.225666 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.225562 2558 scope.go:117] "RemoveContainer" containerID="3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367" Jul 16 22:11:46.234155 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.234141 2558 scope.go:117] "RemoveContainer" containerID="3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367" Jul 16 22:11:46.234412 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:11:46.234393 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367\": container with ID starting with 3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367 not found: ID does not exist" containerID="3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367" Jul 16 22:11:46.234496 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.234418 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367"} err="failed to get container status \"3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367\": rpc error: code = NotFound desc = could not find container \"3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367\": container with ID starting with 3b42ae6072e60693ff45b0f4407e6636a071872c9f55e6cdf0c50ea661ac9367 not found: ID does not exist" Jul 16 22:11:46.246397 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.246372 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn"] Jul 16 22:11:46.250036 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.250015 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-69517-c7754998c-6lqsn"] Jul 16 22:11:46.274602 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.274585 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ae74149-4ea3-4866-82a2-3682c63e516c-openshift-service-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:11:46.274602 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:46.274603 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ae74149-4ea3-4866-82a2-3682c63e516c-proxy-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:11:47.260194 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:47.260162 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae74149-4ea3-4866-82a2-3682c63e516c" path="/var/lib/kubelet/pods/9ae74149-4ea3-4866-82a2-3682c63e516c/volumes" Jul 16 22:11:51.902843 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:51.902807 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82"] Jul 16 22:11:51.903218 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:51.903024 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" podUID="da114ca6-b44d-4a4b-8415-3e17a65a6995" containerName="sequence-graph-77f35" containerID="cri-o://39f955cc92fab75c25be6bbf470d318ee9d272160ae264d9b2856d2946f2de50" gracePeriod=30 Jul 16 22:11:54.457573 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:54.457533 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" podUID="da114ca6-b44d-4a4b-8415-3e17a65a6995" containerName="sequence-graph-77f35" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:11:59.456940 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:11:59.456904 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" podUID="da114ca6-b44d-4a4b-8415-3e17a65a6995" containerName="sequence-graph-77f35" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:12:04.456786 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:04.456749 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" podUID="da114ca6-b44d-4a4b-8415-3e17a65a6995" containerName="sequence-graph-77f35" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:12:04.457200 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:04.456849 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:12:09.457271 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:09.457239 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" podUID="da114ca6-b44d-4a4b-8415-3e17a65a6995" containerName="sequence-graph-77f35" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:12:14.457494 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:14.457458 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" podUID="da114ca6-b44d-4a4b-8415-3e17a65a6995" containerName="sequence-graph-77f35" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:12:19.457456 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:19.457421 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" podUID="da114ca6-b44d-4a4b-8415-3e17a65a6995" containerName="sequence-graph-77f35" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:12:22.360251 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:22.360217 2558 generic.go:356] "Generic (PLEG): container finished" podID="da114ca6-b44d-4a4b-8415-3e17a65a6995" containerID="39f955cc92fab75c25be6bbf470d318ee9d272160ae264d9b2856d2946f2de50" exitCode=0 Jul 16 22:12:22.360606 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:22.360278 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" event={"ID":"da114ca6-b44d-4a4b-8415-3e17a65a6995","Type":"ContainerDied","Data":"39f955cc92fab75c25be6bbf470d318ee9d272160ae264d9b2856d2946f2de50"} Jul 16 22:12:22.546904 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:22.546881 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:12:22.650461 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:22.650392 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da114ca6-b44d-4a4b-8415-3e17a65a6995-openshift-service-ca-bundle\") pod \"da114ca6-b44d-4a4b-8415-3e17a65a6995\" (UID: \"da114ca6-b44d-4a4b-8415-3e17a65a6995\") " Jul 16 22:12:22.650461 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:22.650454 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da114ca6-b44d-4a4b-8415-3e17a65a6995-proxy-tls\") pod \"da114ca6-b44d-4a4b-8415-3e17a65a6995\" (UID: \"da114ca6-b44d-4a4b-8415-3e17a65a6995\") " Jul 16 22:12:22.650804 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:22.650778 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da114ca6-b44d-4a4b-8415-3e17a65a6995-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "da114ca6-b44d-4a4b-8415-3e17a65a6995" (UID: "da114ca6-b44d-4a4b-8415-3e17a65a6995"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:12:22.652547 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:22.652521 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da114ca6-b44d-4a4b-8415-3e17a65a6995-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da114ca6-b44d-4a4b-8415-3e17a65a6995" (UID: "da114ca6-b44d-4a4b-8415-3e17a65a6995"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:12:22.751246 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:22.751214 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da114ca6-b44d-4a4b-8415-3e17a65a6995-proxy-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:12:22.751246 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:22.751240 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da114ca6-b44d-4a4b-8415-3e17a65a6995-openshift-service-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:12:23.364713 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:23.364679 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" event={"ID":"da114ca6-b44d-4a4b-8415-3e17a65a6995","Type":"ContainerDied","Data":"253c8fd4145bdbc8db2316cb4a8d25db6004df9915f2ee7590e477a836d1601b"} Jul 16 22:12:23.365144 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:23.364740 2558 scope.go:117] "RemoveContainer" containerID="39f955cc92fab75c25be6bbf470d318ee9d272160ae264d9b2856d2946f2de50" Jul 16 22:12:23.365144 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:23.364690 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82" Jul 16 22:12:23.379983 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:23.379963 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82"] Jul 16 22:12:23.383822 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:23.383801 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-77f35-5678976b5d-lxn82"] Jul 16 22:12:25.260614 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:25.260583 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da114ca6-b44d-4a4b-8415-3e17a65a6995" path="/var/lib/kubelet/pods/da114ca6-b44d-4a4b-8415-3e17a65a6995/volumes" Jul 16 22:12:26.243191 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.243159 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk"] Jul 16 22:12:26.243509 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.243492 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae74149-4ea3-4866-82a2-3682c63e516c" containerName="switch-graph-69517" Jul 16 22:12:26.243589 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.243512 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae74149-4ea3-4866-82a2-3682c63e516c" containerName="switch-graph-69517" Jul 16 22:12:26.243589 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.243529 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da114ca6-b44d-4a4b-8415-3e17a65a6995" containerName="sequence-graph-77f35" Jul 16 22:12:26.243589 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.243537 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="da114ca6-b44d-4a4b-8415-3e17a65a6995" containerName="sequence-graph-77f35" Jul 16 22:12:26.243753 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.243622 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae74149-4ea3-4866-82a2-3682c63e516c" containerName="switch-graph-69517" Jul 16 22:12:26.243753 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.243635 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="da114ca6-b44d-4a4b-8415-3e17a65a6995" containerName="sequence-graph-77f35" Jul 16 22:12:26.248020 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.248001 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:12:26.251559 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.251536 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-1ab93-kube-rbac-proxy-sar-config\"" Jul 16 22:12:26.251676 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.251539 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-65f4r\"" Jul 16 22:12:26.251917 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.251900 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-1ab93-serving-cert\"" Jul 16 22:12:26.251974 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.251906 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 16 22:12:26.256158 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.256135 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk"] Jul 16 22:12:26.381137 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.381102 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7774d6c0-f8dc-4d18-8782-5d3488aec90f-openshift-service-ca-bundle\") pod \"ensemble-graph-1ab93-5c487f9bf4-zvtmk\" (UID: \"7774d6c0-f8dc-4d18-8782-5d3488aec90f\") " pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:12:26.381137 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.381145 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7774d6c0-f8dc-4d18-8782-5d3488aec90f-proxy-tls\") pod \"ensemble-graph-1ab93-5c487f9bf4-zvtmk\" (UID: \"7774d6c0-f8dc-4d18-8782-5d3488aec90f\") " pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:12:26.482529 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.482504 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7774d6c0-f8dc-4d18-8782-5d3488aec90f-openshift-service-ca-bundle\") pod \"ensemble-graph-1ab93-5c487f9bf4-zvtmk\" (UID: \"7774d6c0-f8dc-4d18-8782-5d3488aec90f\") " pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:12:26.482689 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.482536 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7774d6c0-f8dc-4d18-8782-5d3488aec90f-proxy-tls\") pod \"ensemble-graph-1ab93-5c487f9bf4-zvtmk\" (UID: \"7774d6c0-f8dc-4d18-8782-5d3488aec90f\") " pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:12:26.483202 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.483179 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7774d6c0-f8dc-4d18-8782-5d3488aec90f-openshift-service-ca-bundle\") pod \"ensemble-graph-1ab93-5c487f9bf4-zvtmk\" (UID: \"7774d6c0-f8dc-4d18-8782-5d3488aec90f\") " pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:12:26.485000 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.484970 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7774d6c0-f8dc-4d18-8782-5d3488aec90f-proxy-tls\") pod \"ensemble-graph-1ab93-5c487f9bf4-zvtmk\" (UID: \"7774d6c0-f8dc-4d18-8782-5d3488aec90f\") " pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:12:26.558822 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.558751 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:12:26.685770 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:26.685736 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk"] Jul 16 22:12:26.689616 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:12:26.689586 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7774d6c0_f8dc_4d18_8782_5d3488aec90f.slice/crio-fa730079f56a6ce0773ac2bd838e3d74c1a42e6c945185574339ef84f7722f2d WatchSource:0}: Error finding container fa730079f56a6ce0773ac2bd838e3d74c1a42e6c945185574339ef84f7722f2d: Status 404 returned error can't find the container with id fa730079f56a6ce0773ac2bd838e3d74c1a42e6c945185574339ef84f7722f2d Jul 16 22:12:27.381163 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:27.381126 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" event={"ID":"7774d6c0-f8dc-4d18-8782-5d3488aec90f","Type":"ContainerStarted","Data":"675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63"} Jul 16 22:12:27.381583 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:27.381174 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" event={"ID":"7774d6c0-f8dc-4d18-8782-5d3488aec90f","Type":"ContainerStarted","Data":"fa730079f56a6ce0773ac2bd838e3d74c1a42e6c945185574339ef84f7722f2d"} Jul 16 22:12:27.381583 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:27.381394 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:12:27.398839 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:27.398788 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" podStartSLOduration=1.3987738410000001 podStartE2EDuration="1.398773841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:12:27.398181358 +0000 UTC m=+1348.658091800" watchObservedRunningTime="2026-07-16 22:12:27.398773841 +0000 UTC m=+1348.658684265" Jul 16 22:12:33.390173 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:33.390136 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:12:36.313395 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:36.313366 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk"] Jul 16 22:12:36.313799 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:36.313547 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" podUID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" containerName="ensemble-graph-1ab93" containerID="cri-o://675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63" gracePeriod=30 Jul 16 22:12:38.388920 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:38.388878 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" podUID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" containerName="ensemble-graph-1ab93" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:12:43.389299 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:43.389248 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" podUID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" containerName="ensemble-graph-1ab93" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:12:48.388912 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:48.388851 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" podUID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" containerName="ensemble-graph-1ab93" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:12:48.389293 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:48.389007 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:12:53.388519 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:53.388480 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" podUID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" containerName="ensemble-graph-1ab93" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:12:58.389108 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:12:58.389063 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" podUID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" containerName="ensemble-graph-1ab93" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:13:02.138509 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.138472 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k"] Jul 16 22:13:02.142954 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.142936 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:02.145272 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.145252 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-67446-serving-cert\"" Jul 16 22:13:02.145353 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.145252 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-67446-kube-rbac-proxy-sar-config\"" Jul 16 22:13:02.154196 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.154174 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k"] Jul 16 22:13:02.170590 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.170558 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/329a3eaa-8926-4198-a7d2-142cac7fb99e-openshift-service-ca-bundle\") pod \"sequence-graph-67446-6fbc776749-h7b4k\" (UID: \"329a3eaa-8926-4198-a7d2-142cac7fb99e\") " pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:02.170736 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.170603 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/329a3eaa-8926-4198-a7d2-142cac7fb99e-proxy-tls\") pod \"sequence-graph-67446-6fbc776749-h7b4k\" (UID: \"329a3eaa-8926-4198-a7d2-142cac7fb99e\") " pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:02.271621 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.271585 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/329a3eaa-8926-4198-a7d2-142cac7fb99e-openshift-service-ca-bundle\") pod \"sequence-graph-67446-6fbc776749-h7b4k\" (UID: \"329a3eaa-8926-4198-a7d2-142cac7fb99e\") " pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:02.271802 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.271634 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/329a3eaa-8926-4198-a7d2-142cac7fb99e-proxy-tls\") pod \"sequence-graph-67446-6fbc776749-h7b4k\" (UID: \"329a3eaa-8926-4198-a7d2-142cac7fb99e\") " pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:02.271802 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:13:02.271736 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-67446-serving-cert: secret "sequence-graph-67446-serving-cert" not found Jul 16 22:13:02.271802 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:13:02.271790 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/329a3eaa-8926-4198-a7d2-142cac7fb99e-proxy-tls podName:329a3eaa-8926-4198-a7d2-142cac7fb99e nodeName:}" failed. No retries permitted until 2026-07-16 22:13:02.771773315 +0000 UTC m=+1384.031683716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/329a3eaa-8926-4198-a7d2-142cac7fb99e-proxy-tls") pod "sequence-graph-67446-6fbc776749-h7b4k" (UID: "329a3eaa-8926-4198-a7d2-142cac7fb99e") : secret "sequence-graph-67446-serving-cert" not found Jul 16 22:13:02.272249 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.272230 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/329a3eaa-8926-4198-a7d2-142cac7fb99e-openshift-service-ca-bundle\") pod \"sequence-graph-67446-6fbc776749-h7b4k\" (UID: \"329a3eaa-8926-4198-a7d2-142cac7fb99e\") " pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:02.777760 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.777724 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/329a3eaa-8926-4198-a7d2-142cac7fb99e-proxy-tls\") pod \"sequence-graph-67446-6fbc776749-h7b4k\" (UID: \"329a3eaa-8926-4198-a7d2-142cac7fb99e\") " pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:02.780270 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:02.780249 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/329a3eaa-8926-4198-a7d2-142cac7fb99e-proxy-tls\") pod \"sequence-graph-67446-6fbc776749-h7b4k\" (UID: \"329a3eaa-8926-4198-a7d2-142cac7fb99e\") " pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:03.053043 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:03.052958 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:03.184407 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:03.184319 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k"] Jul 16 22:13:03.186925 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:13:03.186899 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329a3eaa_8926_4198_a7d2_142cac7fb99e.slice/crio-e5ff08ed988a15fe044a6479a6cce4509edd4d9ae8b62719e9ee8d503c94d677 WatchSource:0}: Error finding container e5ff08ed988a15fe044a6479a6cce4509edd4d9ae8b62719e9ee8d503c94d677: Status 404 returned error can't find the container with id e5ff08ed988a15fe044a6479a6cce4509edd4d9ae8b62719e9ee8d503c94d677 Jul 16 22:13:03.388979 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:03.388889 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" podUID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" containerName="ensemble-graph-1ab93" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:13:03.514700 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:03.514666 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" event={"ID":"329a3eaa-8926-4198-a7d2-142cac7fb99e","Type":"ContainerStarted","Data":"c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0"} Jul 16 22:13:03.514887 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:03.514716 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" event={"ID":"329a3eaa-8926-4198-a7d2-142cac7fb99e","Type":"ContainerStarted","Data":"e5ff08ed988a15fe044a6479a6cce4509edd4d9ae8b62719e9ee8d503c94d677"} Jul 16 22:13:03.514887 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:03.514779 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:03.542609 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:03.542558 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" podStartSLOduration=1.542542917 podStartE2EDuration="1.542542917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:13:03.54152347 +0000 UTC m=+1384.801433894" watchObservedRunningTime="2026-07-16 22:13:03.542542917 +0000 UTC m=+1384.802453341" Jul 16 22:13:06.458448 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.458426 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:13:06.507730 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.507703 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7774d6c0-f8dc-4d18-8782-5d3488aec90f-openshift-service-ca-bundle\") pod \"7774d6c0-f8dc-4d18-8782-5d3488aec90f\" (UID: \"7774d6c0-f8dc-4d18-8782-5d3488aec90f\") " Jul 16 22:13:06.507894 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.507753 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7774d6c0-f8dc-4d18-8782-5d3488aec90f-proxy-tls\") pod \"7774d6c0-f8dc-4d18-8782-5d3488aec90f\" (UID: \"7774d6c0-f8dc-4d18-8782-5d3488aec90f\") " Jul 16 22:13:06.508059 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.508038 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7774d6c0-f8dc-4d18-8782-5d3488aec90f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7774d6c0-f8dc-4d18-8782-5d3488aec90f" (UID: "7774d6c0-f8dc-4d18-8782-5d3488aec90f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:13:06.510038 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.510019 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7774d6c0-f8dc-4d18-8782-5d3488aec90f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7774d6c0-f8dc-4d18-8782-5d3488aec90f" (UID: "7774d6c0-f8dc-4d18-8782-5d3488aec90f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:13:06.528187 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.528163 2558 generic.go:356] "Generic (PLEG): container finished" podID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" containerID="675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63" exitCode=0 Jul 16 22:13:06.528305 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.528196 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" event={"ID":"7774d6c0-f8dc-4d18-8782-5d3488aec90f","Type":"ContainerDied","Data":"675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63"} Jul 16 22:13:06.528305 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.528203 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" Jul 16 22:13:06.528305 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.528239 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk" event={"ID":"7774d6c0-f8dc-4d18-8782-5d3488aec90f","Type":"ContainerDied","Data":"fa730079f56a6ce0773ac2bd838e3d74c1a42e6c945185574339ef84f7722f2d"} Jul 16 22:13:06.528305 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.528260 2558 scope.go:117] "RemoveContainer" containerID="675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63" Jul 16 22:13:06.539159 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.539139 2558 scope.go:117] "RemoveContainer" containerID="675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63" Jul 16 22:13:06.539414 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:13:06.539396 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63\": container with ID starting with 675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63 not found: ID does not exist" containerID="675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63" Jul 16 22:13:06.539474 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.539426 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63"} err="failed to get container status \"675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63\": rpc error: code = NotFound desc = could not find container \"675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63\": container with ID starting with 675f0e0dc1483bea418e539286072a30b1ad7c2bbd5eeca53b108f7e1853bc63 not found: ID does not exist" Jul 16 22:13:06.550946 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.550927 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk"] Jul 16 22:13:06.554409 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.554389 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-1ab93-5c487f9bf4-zvtmk"] Jul 16 22:13:06.609062 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.609009 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7774d6c0-f8dc-4d18-8782-5d3488aec90f-openshift-service-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:13:06.609062 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:06.609028 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7774d6c0-f8dc-4d18-8782-5d3488aec90f-proxy-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:13:07.260923 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:07.260893 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" path="/var/lib/kubelet/pods/7774d6c0-f8dc-4d18-8782-5d3488aec90f/volumes" Jul 16 22:13:09.524318 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:09.524292 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:12.244293 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:12.244259 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k"] Jul 16 22:13:12.244733 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:12.244445 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" podUID="329a3eaa-8926-4198-a7d2-142cac7fb99e" containerName="sequence-graph-67446" containerID="cri-o://c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0" gracePeriod=30 Jul 16 22:13:14.523215 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:14.523178 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" podUID="329a3eaa-8926-4198-a7d2-142cac7fb99e" containerName="sequence-graph-67446" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:13:19.522413 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:19.522377 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" podUID="329a3eaa-8926-4198-a7d2-142cac7fb99e" containerName="sequence-graph-67446" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:13:24.522326 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:24.522279 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" podUID="329a3eaa-8926-4198-a7d2-142cac7fb99e" containerName="sequence-graph-67446" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:13:24.522722 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:24.522396 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:29.522500 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:29.522451 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" podUID="329a3eaa-8926-4198-a7d2-142cac7fb99e" containerName="sequence-graph-67446" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:13:34.523270 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:34.523229 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" podUID="329a3eaa-8926-4198-a7d2-142cac7fb99e" containerName="sequence-graph-67446" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:13:39.522656 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:39.522613 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" podUID="329a3eaa-8926-4198-a7d2-142cac7fb99e" containerName="sequence-graph-67446" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:13:42.383014 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.382990 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:42.500288 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.500210 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/329a3eaa-8926-4198-a7d2-142cac7fb99e-proxy-tls\") pod \"329a3eaa-8926-4198-a7d2-142cac7fb99e\" (UID: \"329a3eaa-8926-4198-a7d2-142cac7fb99e\") " Jul 16 22:13:42.500288 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.500286 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/329a3eaa-8926-4198-a7d2-142cac7fb99e-openshift-service-ca-bundle\") pod \"329a3eaa-8926-4198-a7d2-142cac7fb99e\" (UID: \"329a3eaa-8926-4198-a7d2-142cac7fb99e\") " Jul 16 22:13:42.500631 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.500608 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329a3eaa-8926-4198-a7d2-142cac7fb99e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "329a3eaa-8926-4198-a7d2-142cac7fb99e" (UID: "329a3eaa-8926-4198-a7d2-142cac7fb99e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:13:42.502448 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.502430 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329a3eaa-8926-4198-a7d2-142cac7fb99e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "329a3eaa-8926-4198-a7d2-142cac7fb99e" (UID: "329a3eaa-8926-4198-a7d2-142cac7fb99e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:13:42.601372 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.601345 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/329a3eaa-8926-4198-a7d2-142cac7fb99e-proxy-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:13:42.601372 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.601369 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/329a3eaa-8926-4198-a7d2-142cac7fb99e-openshift-service-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:13:42.656660 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.656624 2558 generic.go:356] "Generic (PLEG): container finished" podID="329a3eaa-8926-4198-a7d2-142cac7fb99e" containerID="c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0" exitCode=0 Jul 16 22:13:42.656794 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.656670 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" Jul 16 22:13:42.656794 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.656717 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" event={"ID":"329a3eaa-8926-4198-a7d2-142cac7fb99e","Type":"ContainerDied","Data":"c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0"} Jul 16 22:13:42.656794 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.656768 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k" event={"ID":"329a3eaa-8926-4198-a7d2-142cac7fb99e","Type":"ContainerDied","Data":"e5ff08ed988a15fe044a6479a6cce4509edd4d9ae8b62719e9ee8d503c94d677"} Jul 16 22:13:42.656794 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.656789 2558 scope.go:117] "RemoveContainer" containerID="c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0" Jul 16 22:13:42.670608 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.670589 2558 scope.go:117] "RemoveContainer" containerID="c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0" Jul 16 22:13:42.670908 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:13:42.670881 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0\": container with ID starting with c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0 not found: ID does not exist" containerID="c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0" Jul 16 22:13:42.670956 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.670920 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0"} err="failed to get container status \"c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0\": rpc error: code = NotFound desc = could not find container \"c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0\": container with ID starting with c39b4e017bf99daf554b393769ee894051a6343f5d4928b5e6b8a80ae2c987b0 not found: ID does not exist" Jul 16 22:13:42.681340 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.681316 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k"] Jul 16 22:13:42.685056 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:42.685038 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67446-6fbc776749-h7b4k"] Jul 16 22:13:43.260270 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:43.260237 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="329a3eaa-8926-4198-a7d2-142cac7fb99e" path="/var/lib/kubelet/pods/329a3eaa-8926-4198-a7d2-142cac7fb99e/volumes" Jul 16 22:13:46.615921 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.615886 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg"] Jul 16 22:13:46.616381 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.616290 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" containerName="ensemble-graph-1ab93" Jul 16 22:13:46.616381 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.616310 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" containerName="ensemble-graph-1ab93" Jul 16 22:13:46.616381 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.616323 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329a3eaa-8926-4198-a7d2-142cac7fb99e" containerName="sequence-graph-67446" Jul 16 22:13:46.616381 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.616332 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="329a3eaa-8926-4198-a7d2-142cac7fb99e" containerName="sequence-graph-67446" Jul 16 22:13:46.616579 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.616415 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="329a3eaa-8926-4198-a7d2-142cac7fb99e" containerName="sequence-graph-67446" Jul 16 22:13:46.616579 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.616427 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7774d6c0-f8dc-4d18-8782-5d3488aec90f" containerName="ensemble-graph-1ab93" Jul 16 22:13:46.620764 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.620744 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:13:46.623614 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.623593 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-65f4r\"" Jul 16 22:13:46.623715 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.623644 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c1850-serving-cert\"" Jul 16 22:13:46.623889 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.623848 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-c1850-kube-rbac-proxy-sar-config\"" Jul 16 22:13:46.624249 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.624235 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 16 22:13:46.631188 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.631169 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg"] Jul 16 22:13:46.735747 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.735717 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-openshift-service-ca-bundle\") pod \"ensemble-graph-c1850-7f4cf9486c-vknjg\" (UID: \"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7\") " pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:13:46.735936 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.735769 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-proxy-tls\") pod \"ensemble-graph-c1850-7f4cf9486c-vknjg\" (UID: \"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7\") " pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:13:46.837233 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.837198 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-openshift-service-ca-bundle\") pod \"ensemble-graph-c1850-7f4cf9486c-vknjg\" (UID: \"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7\") " pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:13:46.837233 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.837238 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-proxy-tls\") pod \"ensemble-graph-c1850-7f4cf9486c-vknjg\" (UID: \"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7\") " pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:13:46.837818 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.837794 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-openshift-service-ca-bundle\") pod \"ensemble-graph-c1850-7f4cf9486c-vknjg\" (UID: \"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7\") " pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:13:46.839769 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.839742 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-proxy-tls\") pod \"ensemble-graph-c1850-7f4cf9486c-vknjg\" (UID: \"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7\") " pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:13:46.931470 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:46.931449 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:13:47.081741 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:47.081713 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg"] Jul 16 22:13:47.084828 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:13:47.084801 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d8e27b_0eba_4c02_b3b0_f50f5ddcedc7.slice/crio-a987aedc34e53e6a9077bca12bd4216013490c9063e84104975070eb34f5baa8 WatchSource:0}: Error finding container a987aedc34e53e6a9077bca12bd4216013490c9063e84104975070eb34f5baa8: Status 404 returned error can't find the container with id a987aedc34e53e6a9077bca12bd4216013490c9063e84104975070eb34f5baa8 Jul 16 22:13:47.676380 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:47.676345 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" event={"ID":"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7","Type":"ContainerStarted","Data":"53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4"} Jul 16 22:13:47.676800 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:47.676392 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" event={"ID":"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7","Type":"ContainerStarted","Data":"a987aedc34e53e6a9077bca12bd4216013490c9063e84104975070eb34f5baa8"} Jul 16 22:13:47.676800 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:47.676422 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:13:47.706712 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:47.706671 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" podStartSLOduration=1.706656583 podStartE2EDuration="1.706656583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:13:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:13:47.706002609 +0000 UTC m=+1428.965913033" watchObservedRunningTime="2026-07-16 22:13:47.706656583 +0000 UTC m=+1428.966567007" Jul 16 22:13:53.686465 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:13:53.686394 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:14:22.515654 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:22.515621 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5"] Jul 16 22:14:22.520634 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:22.520614 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:14:22.523081 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:22.523062 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a3a91-serving-cert\"" Jul 16 22:14:22.523081 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:22.523071 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-a3a91-kube-rbac-proxy-sar-config\"" Jul 16 22:14:22.526766 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:22.526747 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5"] Jul 16 22:14:22.611959 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:22.611910 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-openshift-service-ca-bundle\") pod \"sequence-graph-a3a91-54fb849f7b-5fgx5\" (UID: \"afcdd5f1-6428-40cd-bd8c-851217ba3a6b\") " pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:14:22.612119 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:22.612028 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-proxy-tls\") pod \"sequence-graph-a3a91-54fb849f7b-5fgx5\" (UID: \"afcdd5f1-6428-40cd-bd8c-851217ba3a6b\") " pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:14:22.713080 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:22.713045 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-openshift-service-ca-bundle\") pod \"sequence-graph-a3a91-54fb849f7b-5fgx5\" (UID: \"afcdd5f1-6428-40cd-bd8c-851217ba3a6b\") " pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:14:22.713239 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:22.713091 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-proxy-tls\") pod \"sequence-graph-a3a91-54fb849f7b-5fgx5\" (UID: \"afcdd5f1-6428-40cd-bd8c-851217ba3a6b\") " pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:14:22.713239 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:14:22.713208 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-a3a91-serving-cert: secret "sequence-graph-a3a91-serving-cert" not found Jul 16 22:14:22.713325 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:14:22.713275 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-proxy-tls podName:afcdd5f1-6428-40cd-bd8c-851217ba3a6b nodeName:}" failed. No retries permitted until 2026-07-16 22:14:23.21325778 +0000 UTC m=+1464.473168182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-proxy-tls") pod "sequence-graph-a3a91-54fb849f7b-5fgx5" (UID: "afcdd5f1-6428-40cd-bd8c-851217ba3a6b") : secret "sequence-graph-a3a91-serving-cert" not found Jul 16 22:14:22.713844 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:22.713820 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-openshift-service-ca-bundle\") pod \"sequence-graph-a3a91-54fb849f7b-5fgx5\" (UID: \"afcdd5f1-6428-40cd-bd8c-851217ba3a6b\") " pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:14:23.216276 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:23.216239 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-proxy-tls\") pod \"sequence-graph-a3a91-54fb849f7b-5fgx5\" (UID: \"afcdd5f1-6428-40cd-bd8c-851217ba3a6b\") " pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:14:23.218997 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:23.218968 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-proxy-tls\") pod \"sequence-graph-a3a91-54fb849f7b-5fgx5\" (UID: \"afcdd5f1-6428-40cd-bd8c-851217ba3a6b\") " pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:14:23.432487 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:23.432460 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:14:23.561291 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:23.561244 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5"] Jul 16 22:14:23.563485 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:14:23.563460 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafcdd5f1_6428_40cd_bd8c_851217ba3a6b.slice/crio-8e5ca6ba12b404e5cc04aedcf796944ffc718ffee6dd08f0a97c0c8129b0ec2a WatchSource:0}: Error finding container 8e5ca6ba12b404e5cc04aedcf796944ffc718ffee6dd08f0a97c0c8129b0ec2a: Status 404 returned error can't find the container with id 8e5ca6ba12b404e5cc04aedcf796944ffc718ffee6dd08f0a97c0c8129b0ec2a Jul 16 22:14:23.814651 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:23.814560 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" event={"ID":"afcdd5f1-6428-40cd-bd8c-851217ba3a6b","Type":"ContainerStarted","Data":"58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c"} Jul 16 22:14:23.814651 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:23.814637 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" event={"ID":"afcdd5f1-6428-40cd-bd8c-851217ba3a6b","Type":"ContainerStarted","Data":"8e5ca6ba12b404e5cc04aedcf796944ffc718ffee6dd08f0a97c0c8129b0ec2a"} Jul 16 22:14:23.814883 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:23.814672 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:14:23.833515 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:23.833468 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" podStartSLOduration=1.833454444 podStartE2EDuration="1.833454444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:14:23.833078219 +0000 UTC m=+1465.092988645" watchObservedRunningTime="2026-07-16 22:14:23.833454444 +0000 UTC m=+1465.093364867" Jul 16 22:14:29.824086 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:14:29.824061 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:22:01.208507 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:01.208471 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg"] Jul 16 22:22:01.209060 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:01.208748 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" podUID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" containerName="ensemble-graph-c1850" containerID="cri-o://53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4" gracePeriod=30 Jul 16 22:22:03.684831 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:03.684789 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" podUID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" containerName="ensemble-graph-c1850" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:22:08.684685 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:08.684647 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" podUID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" containerName="ensemble-graph-c1850" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:22:13.685395 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:13.685348 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" podUID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" containerName="ensemble-graph-c1850" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:22:13.685845 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:13.685475 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:22:18.685439 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:18.685393 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" podUID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" containerName="ensemble-graph-c1850" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:22:23.685556 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:23.685519 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" podUID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" containerName="ensemble-graph-c1850" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:22:28.685506 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:28.685469 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" podUID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" containerName="ensemble-graph-c1850" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:22:31.359830 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.359806 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:22:31.436119 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.436082 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-openshift-service-ca-bundle\") pod \"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7\" (UID: \"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7\") " Jul 16 22:22:31.436295 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.436129 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-proxy-tls\") pod \"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7\" (UID: \"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7\") " Jul 16 22:22:31.436467 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.436441 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" (UID: "c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:22:31.438416 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.438396 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" (UID: "c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:22:31.537641 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.537557 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-openshift-service-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:22:31.537641 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.537588 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7-proxy-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:22:31.588509 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.588473 2558 generic.go:356] "Generic (PLEG): container finished" podID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" containerID="53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4" exitCode=0 Jul 16 22:22:31.588690 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.588520 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" Jul 16 22:22:31.588690 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.588557 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" event={"ID":"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7","Type":"ContainerDied","Data":"53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4"} Jul 16 22:22:31.588690 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.588631 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg" event={"ID":"c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7","Type":"ContainerDied","Data":"a987aedc34e53e6a9077bca12bd4216013490c9063e84104975070eb34f5baa8"} Jul 16 22:22:31.588690 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.588652 2558 scope.go:117] "RemoveContainer" containerID="53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4" Jul 16 22:22:31.597078 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.597057 2558 scope.go:117] "RemoveContainer" containerID="53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4" Jul 16 22:22:31.597334 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:22:31.597316 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4\": container with ID starting with 53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4 not found: ID does not exist" containerID="53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4" Jul 16 22:22:31.597398 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.597342 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4"} err="failed to get container status \"53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4\": rpc error: code = NotFound desc = could not find container \"53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4\": container with ID starting with 53270074f2838fe982965e4090444007465c06553808240bd7d3b8351e0ad0b4 not found: ID does not exist" Jul 16 22:22:31.610309 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.610286 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg"] Jul 16 22:22:31.614615 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:31.614592 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-c1850-7f4cf9486c-vknjg"] Jul 16 22:22:33.260615 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:33.260583 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" path="/var/lib/kubelet/pods/c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7/volumes" Jul 16 22:22:37.211548 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:37.211517 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5"] Jul 16 22:22:37.212130 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:37.211757 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" podUID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" containerName="sequence-graph-a3a91" containerID="cri-o://58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c" gracePeriod=30 Jul 16 22:22:39.823112 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:39.823074 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" podUID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" containerName="sequence-graph-a3a91" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:22:44.823142 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:44.823099 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" podUID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" containerName="sequence-graph-a3a91" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:22:49.823099 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:49.823060 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" podUID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" containerName="sequence-graph-a3a91" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:22:49.823564 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:49.823159 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:22:54.822928 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:54.822889 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" podUID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" containerName="sequence-graph-a3a91" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:22:59.823040 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:22:59.823001 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" podUID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" containerName="sequence-graph-a3a91" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:23:04.822410 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:04.822373 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" podUID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" containerName="sequence-graph-a3a91" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:23:07.358451 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.358429 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:23:07.424323 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.424295 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-openshift-service-ca-bundle\") pod \"afcdd5f1-6428-40cd-bd8c-851217ba3a6b\" (UID: \"afcdd5f1-6428-40cd-bd8c-851217ba3a6b\") " Jul 16 22:23:07.424481 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.424360 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-proxy-tls\") pod \"afcdd5f1-6428-40cd-bd8c-851217ba3a6b\" (UID: \"afcdd5f1-6428-40cd-bd8c-851217ba3a6b\") " Jul 16 22:23:07.424662 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.424633 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "afcdd5f1-6428-40cd-bd8c-851217ba3a6b" (UID: "afcdd5f1-6428-40cd-bd8c-851217ba3a6b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:23:07.426380 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.426359 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afcdd5f1-6428-40cd-bd8c-851217ba3a6b" (UID: "afcdd5f1-6428-40cd-bd8c-851217ba3a6b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:23:07.525243 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.525165 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-openshift-service-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:23:07.525243 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.525192 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afcdd5f1-6428-40cd-bd8c-851217ba3a6b-proxy-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:23:07.718238 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.718206 2558 generic.go:356] "Generic (PLEG): container finished" podID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" containerID="58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c" exitCode=0 Jul 16 22:23:07.718395 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.718248 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" event={"ID":"afcdd5f1-6428-40cd-bd8c-851217ba3a6b","Type":"ContainerDied","Data":"58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c"} Jul 16 22:23:07.718395 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.718288 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" event={"ID":"afcdd5f1-6428-40cd-bd8c-851217ba3a6b","Type":"ContainerDied","Data":"8e5ca6ba12b404e5cc04aedcf796944ffc718ffee6dd08f0a97c0c8129b0ec2a"} Jul 16 22:23:07.718395 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.718308 2558 scope.go:117] "RemoveContainer" containerID="58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c" Jul 16 22:23:07.718395 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.718252 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5" Jul 16 22:23:07.727442 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.727425 2558 scope.go:117] "RemoveContainer" containerID="58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c" Jul 16 22:23:07.727676 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:23:07.727661 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c\": container with ID starting with 58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c not found: ID does not exist" containerID="58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c" Jul 16 22:23:07.727715 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.727684 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c"} err="failed to get container status \"58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c\": rpc error: code = NotFound desc = could not find container \"58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c\": container with ID starting with 58345265273c7c3f52d6dd664b92bb616a0d8451ffe82727ae8daac8b4f6cb4c not found: ID does not exist" Jul 16 22:23:07.745038 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.745014 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5"] Jul 16 22:23:07.747190 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:07.747172 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-a3a91-54fb849f7b-5fgx5"] Jul 16 22:23:09.261431 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:09.261388 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" path="/var/lib/kubelet/pods/afcdd5f1-6428-40cd-bd8c-851217ba3a6b/volumes" Jul 16 22:23:11.518279 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.518196 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j"] Jul 16 22:23:11.518658 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.518600 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" containerName="ensemble-graph-c1850" Jul 16 22:23:11.518658 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.518615 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" containerName="ensemble-graph-c1850" Jul 16 22:23:11.518658 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.518655 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" containerName="sequence-graph-a3a91" Jul 16 22:23:11.518778 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.518663 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" containerName="sequence-graph-a3a91" Jul 16 22:23:11.518778 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.518723 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="afcdd5f1-6428-40cd-bd8c-851217ba3a6b" containerName="sequence-graph-a3a91" Jul 16 22:23:11.518778 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.518733 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d8e27b-0eba-4c02-b3b0-f50f5ddcedc7" containerName="ensemble-graph-c1850" Jul 16 22:23:11.523238 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.523218 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:11.526490 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.526465 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-a3d07-kube-rbac-proxy-sar-config\"" Jul 16 22:23:11.526625 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.526612 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-65f4r\"" Jul 16 22:23:11.526894 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.526851 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-a3d07-serving-cert\"" Jul 16 22:23:11.527282 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.527265 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 16 22:23:11.535489 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.535470 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j"] Jul 16 22:23:11.658785 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.658747 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a77efe-01d3-443e-a0f7-82066068fcec-proxy-tls\") pod \"splitter-graph-a3d07-5bd98c69cb-ffz2j\" (UID: \"14a77efe-01d3-443e-a0f7-82066068fcec\") " pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:11.658976 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.658817 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a77efe-01d3-443e-a0f7-82066068fcec-openshift-service-ca-bundle\") pod \"splitter-graph-a3d07-5bd98c69cb-ffz2j\" (UID: \"14a77efe-01d3-443e-a0f7-82066068fcec\") " pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:11.759325 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.759277 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a77efe-01d3-443e-a0f7-82066068fcec-openshift-service-ca-bundle\") pod \"splitter-graph-a3d07-5bd98c69cb-ffz2j\" (UID: \"14a77efe-01d3-443e-a0f7-82066068fcec\") " pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:11.759523 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.759345 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a77efe-01d3-443e-a0f7-82066068fcec-proxy-tls\") pod \"splitter-graph-a3d07-5bd98c69cb-ffz2j\" (UID: \"14a77efe-01d3-443e-a0f7-82066068fcec\") " pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:11.759523 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:23:11.759432 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-a3d07-serving-cert: secret "splitter-graph-a3d07-serving-cert" not found Jul 16 22:23:11.759523 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:23:11.759488 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14a77efe-01d3-443e-a0f7-82066068fcec-proxy-tls podName:14a77efe-01d3-443e-a0f7-82066068fcec nodeName:}" failed. No retries permitted until 2026-07-16 22:23:12.259471421 +0000 UTC m=+1993.519381823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/14a77efe-01d3-443e-a0f7-82066068fcec-proxy-tls") pod "splitter-graph-a3d07-5bd98c69cb-ffz2j" (UID: "14a77efe-01d3-443e-a0f7-82066068fcec") : secret "splitter-graph-a3d07-serving-cert" not found Jul 16 22:23:11.759958 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:11.759940 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a77efe-01d3-443e-a0f7-82066068fcec-openshift-service-ca-bundle\") pod \"splitter-graph-a3d07-5bd98c69cb-ffz2j\" (UID: \"14a77efe-01d3-443e-a0f7-82066068fcec\") " pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:12.263174 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:12.263138 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a77efe-01d3-443e-a0f7-82066068fcec-proxy-tls\") pod \"splitter-graph-a3d07-5bd98c69cb-ffz2j\" (UID: \"14a77efe-01d3-443e-a0f7-82066068fcec\") " pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:12.265691 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:12.265663 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a77efe-01d3-443e-a0f7-82066068fcec-proxy-tls\") pod \"splitter-graph-a3d07-5bd98c69cb-ffz2j\" (UID: \"14a77efe-01d3-443e-a0f7-82066068fcec\") " pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:12.434775 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:12.434712 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:12.578094 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:12.578061 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j"] Jul 16 22:23:12.586294 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:23:12.586263 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a77efe_01d3_443e_a0f7_82066068fcec.slice/crio-445761682e9ced817c616cfc1e206a425843ef3006606a1c573193c209ebeccc WatchSource:0}: Error finding container 445761682e9ced817c616cfc1e206a425843ef3006606a1c573193c209ebeccc: Status 404 returned error can't find the container with id 445761682e9ced817c616cfc1e206a425843ef3006606a1c573193c209ebeccc Jul 16 22:23:12.741327 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:12.741285 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" event={"ID":"14a77efe-01d3-443e-a0f7-82066068fcec","Type":"ContainerStarted","Data":"d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f"} Jul 16 22:23:12.741491 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:12.741338 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" event={"ID":"14a77efe-01d3-443e-a0f7-82066068fcec","Type":"ContainerStarted","Data":"445761682e9ced817c616cfc1e206a425843ef3006606a1c573193c209ebeccc"} Jul 16 22:23:12.741491 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:12.741368 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:12.758413 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:12.758357 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" podStartSLOduration=1.758338336 podStartE2EDuration="1.758338336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:23:12.758090321 +0000 UTC m=+1994.018000781" watchObservedRunningTime="2026-07-16 22:23:12.758338336 +0000 UTC m=+1994.018248765" Jul 16 22:23:18.751373 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:18.751344 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:21.680404 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:21.680369 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j"] Jul 16 22:23:21.680871 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:21.680586 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" podUID="14a77efe-01d3-443e-a0f7-82066068fcec" containerName="splitter-graph-a3d07" containerID="cri-o://d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f" gracePeriod=30 Jul 16 22:23:23.749164 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:23.749128 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" podUID="14a77efe-01d3-443e-a0f7-82066068fcec" containerName="splitter-graph-a3d07" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:23:28.749116 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:28.749078 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" podUID="14a77efe-01d3-443e-a0f7-82066068fcec" containerName="splitter-graph-a3d07" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:23:33.748816 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:33.748781 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" podUID="14a77efe-01d3-443e-a0f7-82066068fcec" containerName="splitter-graph-a3d07" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:23:33.749267 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:33.748930 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:38.749498 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:38.749454 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" podUID="14a77efe-01d3-443e-a0f7-82066068fcec" containerName="splitter-graph-a3d07" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:23:43.749401 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:43.749354 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" podUID="14a77efe-01d3-443e-a0f7-82066068fcec" containerName="splitter-graph-a3d07" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:23:47.516442 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.516402 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc"] Jul 16 22:23:47.519692 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.519675 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:23:47.523134 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.523115 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-93462-serving-cert\"" Jul 16 22:23:47.523381 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.523362 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-93462-kube-rbac-proxy-sar-config\"" Jul 16 22:23:47.538111 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.538088 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc"] Jul 16 22:23:47.619729 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.619701 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b417db4-3fb1-4246-b106-5113a3334c64-openshift-service-ca-bundle\") pod \"switch-graph-93462-6584468b49-vtflc\" (UID: \"5b417db4-3fb1-4246-b106-5113a3334c64\") " pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:23:47.619918 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.619752 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b417db4-3fb1-4246-b106-5113a3334c64-proxy-tls\") pod \"switch-graph-93462-6584468b49-vtflc\" (UID: \"5b417db4-3fb1-4246-b106-5113a3334c64\") " pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:23:47.721294 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.721261 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b417db4-3fb1-4246-b106-5113a3334c64-openshift-service-ca-bundle\") pod \"switch-graph-93462-6584468b49-vtflc\" (UID: \"5b417db4-3fb1-4246-b106-5113a3334c64\") " pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:23:47.721464 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.721319 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b417db4-3fb1-4246-b106-5113a3334c64-proxy-tls\") pod \"switch-graph-93462-6584468b49-vtflc\" (UID: \"5b417db4-3fb1-4246-b106-5113a3334c64\") " pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:23:47.721911 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.721889 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b417db4-3fb1-4246-b106-5113a3334c64-openshift-service-ca-bundle\") pod \"switch-graph-93462-6584468b49-vtflc\" (UID: \"5b417db4-3fb1-4246-b106-5113a3334c64\") " pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:23:47.723882 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.723828 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b417db4-3fb1-4246-b106-5113a3334c64-proxy-tls\") pod \"switch-graph-93462-6584468b49-vtflc\" (UID: \"5b417db4-3fb1-4246-b106-5113a3334c64\") " pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:23:47.829713 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.829636 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:23:47.961540 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:47.961514 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc"] Jul 16 22:23:47.964135 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:23:47.964107 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b417db4_3fb1_4246_b106_5113a3334c64.slice/crio-7e2c64d80809d262a42aa23006dc1b585e2fb057e058c8944568ab9d143bc259 WatchSource:0}: Error finding container 7e2c64d80809d262a42aa23006dc1b585e2fb057e058c8944568ab9d143bc259: Status 404 returned error can't find the container with id 7e2c64d80809d262a42aa23006dc1b585e2fb057e058c8944568ab9d143bc259 Jul 16 22:23:48.749288 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:48.749239 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" podUID="14a77efe-01d3-443e-a0f7-82066068fcec" containerName="splitter-graph-a3d07" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:23:48.871071 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:48.871040 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" event={"ID":"5b417db4-3fb1-4246-b106-5113a3334c64","Type":"ContainerStarted","Data":"a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01"} Jul 16 22:23:48.871252 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:48.871117 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" event={"ID":"5b417db4-3fb1-4246-b106-5113a3334c64","Type":"ContainerStarted","Data":"7e2c64d80809d262a42aa23006dc1b585e2fb057e058c8944568ab9d143bc259"} Jul 16 22:23:48.871252 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:48.871153 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:23:48.889352 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:48.889307 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" podStartSLOduration=1.8892929189999998 podStartE2EDuration="1.889292919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:23:48.888180071 +0000 UTC m=+2030.148090496" watchObservedRunningTime="2026-07-16 22:23:48.889292919 +0000 UTC m=+2030.149203342" Jul 16 22:23:51.827119 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.827098 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:51.854048 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.854017 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a77efe-01d3-443e-a0f7-82066068fcec-openshift-service-ca-bundle\") pod \"14a77efe-01d3-443e-a0f7-82066068fcec\" (UID: \"14a77efe-01d3-443e-a0f7-82066068fcec\") " Jul 16 22:23:51.854178 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.854054 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a77efe-01d3-443e-a0f7-82066068fcec-proxy-tls\") pod \"14a77efe-01d3-443e-a0f7-82066068fcec\" (UID: \"14a77efe-01d3-443e-a0f7-82066068fcec\") " Jul 16 22:23:51.854342 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.854317 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14a77efe-01d3-443e-a0f7-82066068fcec-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "14a77efe-01d3-443e-a0f7-82066068fcec" (UID: "14a77efe-01d3-443e-a0f7-82066068fcec"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:23:51.856102 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.856078 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14a77efe-01d3-443e-a0f7-82066068fcec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14a77efe-01d3-443e-a0f7-82066068fcec" (UID: "14a77efe-01d3-443e-a0f7-82066068fcec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:23:51.882373 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.882342 2558 generic.go:356] "Generic (PLEG): container finished" podID="14a77efe-01d3-443e-a0f7-82066068fcec" containerID="d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f" exitCode=0 Jul 16 22:23:51.882459 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.882388 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" Jul 16 22:23:51.882459 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.882412 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" event={"ID":"14a77efe-01d3-443e-a0f7-82066068fcec","Type":"ContainerDied","Data":"d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f"} Jul 16 22:23:51.882528 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.882467 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j" event={"ID":"14a77efe-01d3-443e-a0f7-82066068fcec","Type":"ContainerDied","Data":"445761682e9ced817c616cfc1e206a425843ef3006606a1c573193c209ebeccc"} Jul 16 22:23:51.882528 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.882494 2558 scope.go:117] "RemoveContainer" containerID="d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f" Jul 16 22:23:51.890768 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.890753 2558 scope.go:117] "RemoveContainer" containerID="d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f" Jul 16 22:23:51.891013 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:23:51.890998 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f\": container with ID starting with d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f not found: ID does not exist" containerID="d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f" Jul 16 22:23:51.891081 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.891019 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f"} err="failed to get container status \"d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f\": rpc error: code = NotFound desc = could not find container \"d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f\": container with ID starting with d7088adae0305c356f0459d5557ebe6c854b14dbdd4c94b0523b8a7a79522d7f not found: ID does not exist" Jul 16 22:23:51.907411 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.907390 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j"] Jul 16 22:23:51.912766 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.912748 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-a3d07-5bd98c69cb-ffz2j"] Jul 16 22:23:51.954906 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.954833 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14a77efe-01d3-443e-a0f7-82066068fcec-openshift-service-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:23:51.954906 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:51.954871 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14a77efe-01d3-443e-a0f7-82066068fcec-proxy-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:23:53.260762 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:53.260727 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14a77efe-01d3-443e-a0f7-82066068fcec" path="/var/lib/kubelet/pods/14a77efe-01d3-443e-a0f7-82066068fcec/volumes" Jul 16 22:23:54.880265 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:23:54.880227 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:24:21.966673 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:21.966588 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w"] Jul 16 22:24:21.967218 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:21.967055 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14a77efe-01d3-443e-a0f7-82066068fcec" containerName="splitter-graph-a3d07" Jul 16 22:24:21.967218 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:21.967073 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="14a77efe-01d3-443e-a0f7-82066068fcec" containerName="splitter-graph-a3d07" Jul 16 22:24:21.967218 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:21.967144 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="14a77efe-01d3-443e-a0f7-82066068fcec" containerName="splitter-graph-a3d07" Jul 16 22:24:21.975276 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:21.975253 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:24:21.978948 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:21.978927 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-76a97-serving-cert\"" Jul 16 22:24:21.979047 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:21.978927 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-76a97-kube-rbac-proxy-sar-config\"" Jul 16 22:24:21.982562 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:21.982545 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w"] Jul 16 22:24:22.101610 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:22.101571 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-proxy-tls\") pod \"splitter-graph-76a97-8669877574-rbn4w\" (UID: \"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed\") " pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:24:22.101771 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:22.101618 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-openshift-service-ca-bundle\") pod \"splitter-graph-76a97-8669877574-rbn4w\" (UID: \"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed\") " pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:24:22.202479 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:22.202439 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-proxy-tls\") pod \"splitter-graph-76a97-8669877574-rbn4w\" (UID: \"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed\") " pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:24:22.202684 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:22.202495 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-openshift-service-ca-bundle\") pod \"splitter-graph-76a97-8669877574-rbn4w\" (UID: \"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed\") " pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:24:22.202684 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:24:22.202621 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-76a97-serving-cert: secret "splitter-graph-76a97-serving-cert" not found Jul 16 22:24:22.202808 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:24:22.202702 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-proxy-tls podName:e2e1c182-96b5-478f-a3a5-b5bef18bc6ed nodeName:}" failed. No retries permitted until 2026-07-16 22:24:22.702684343 +0000 UTC m=+2063.962594745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-proxy-tls") pod "splitter-graph-76a97-8669877574-rbn4w" (UID: "e2e1c182-96b5-478f-a3a5-b5bef18bc6ed") : secret "splitter-graph-76a97-serving-cert" not found Jul 16 22:24:22.203260 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:22.203236 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-openshift-service-ca-bundle\") pod \"splitter-graph-76a97-8669877574-rbn4w\" (UID: \"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed\") " pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:24:22.706898 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:22.706841 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-proxy-tls\") pod \"splitter-graph-76a97-8669877574-rbn4w\" (UID: \"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed\") " pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:24:22.709356 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:22.709335 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-proxy-tls\") pod \"splitter-graph-76a97-8669877574-rbn4w\" (UID: \"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed\") " pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:24:22.885827 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:22.885781 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:24:23.014179 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:23.014129 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w"] Jul 16 22:24:23.016712 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:24:23.016687 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e1c182_96b5_478f_a3a5_b5bef18bc6ed.slice/crio-a05127157a3d5d7692937d1cf087543d2c0956bc7a20a5e2707703067dd141cf WatchSource:0}: Error finding container a05127157a3d5d7692937d1cf087543d2c0956bc7a20a5e2707703067dd141cf: Status 404 returned error can't find the container with id a05127157a3d5d7692937d1cf087543d2c0956bc7a20a5e2707703067dd141cf Jul 16 22:24:24.004225 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:24.004187 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" event={"ID":"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed","Type":"ContainerStarted","Data":"1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70"} Jul 16 22:24:24.004413 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:24.004250 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" event={"ID":"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed","Type":"ContainerStarted","Data":"a05127157a3d5d7692937d1cf087543d2c0956bc7a20a5e2707703067dd141cf"} Jul 16 22:24:24.004413 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:24.004292 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:24:24.031079 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:24.031028 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" podStartSLOduration=3.031013043 podStartE2EDuration="3.031013043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:24:24.029174637 +0000 UTC m=+2065.289085061" watchObservedRunningTime="2026-07-16 22:24:24.031013043 +0000 UTC m=+2065.290923467" Jul 16 22:24:30.013985 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:24:30.013954 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:32:36.718554 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:32:36.718523 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w"] Jul 16 22:32:36.720160 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:32:36.718717 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" podUID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" containerName="splitter-graph-76a97" containerID="cri-o://1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70" gracePeriod=30 Jul 16 22:32:40.011890 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:32:40.011833 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" podUID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" containerName="splitter-graph-76a97" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:32:45.011623 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:32:45.011568 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" podUID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" containerName="splitter-graph-76a97" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:32:50.012591 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:32:50.012551 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" podUID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" containerName="splitter-graph-76a97" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:32:50.013064 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:32:50.012655 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:32:55.012276 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:32:55.012234 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" podUID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" containerName="splitter-graph-76a97" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:33:00.011821 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:00.011781 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" podUID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" containerName="splitter-graph-76a97" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:33:05.012221 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:05.012182 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" podUID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" containerName="splitter-graph-76a97" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:33:06.858137 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.858114 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:33:06.916328 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.916299 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-openshift-service-ca-bundle\") pod \"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed\" (UID: \"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed\") " Jul 16 22:33:06.916533 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.916365 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-proxy-tls\") pod \"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed\" (UID: \"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed\") " Jul 16 22:33:06.916702 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.916681 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" (UID: "e2e1c182-96b5-478f-a3a5-b5bef18bc6ed"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:33:06.917666 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.917632 2558 generic.go:356] "Generic (PLEG): container finished" podID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" containerID="1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70" exitCode=0 Jul 16 22:33:06.917798 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.917673 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" Jul 16 22:33:06.917798 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.917689 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" event={"ID":"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed","Type":"ContainerDied","Data":"1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70"} Jul 16 22:33:06.917798 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.917760 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w" event={"ID":"e2e1c182-96b5-478f-a3a5-b5bef18bc6ed","Type":"ContainerDied","Data":"a05127157a3d5d7692937d1cf087543d2c0956bc7a20a5e2707703067dd141cf"} Jul 16 22:33:06.917798 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.917792 2558 scope.go:117] "RemoveContainer" containerID="1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70" Jul 16 22:33:06.919076 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.918960 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" (UID: "e2e1c182-96b5-478f-a3a5-b5bef18bc6ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:33:06.928522 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.928499 2558 scope.go:117] "RemoveContainer" containerID="1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70" Jul 16 22:33:06.928995 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:33:06.928974 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70\": container with ID starting with 1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70 not found: ID does not exist" containerID="1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70" Jul 16 22:33:06.929106 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:06.929006 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70"} err="failed to get container status \"1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70\": rpc error: code = NotFound desc = could not find container \"1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70\": container with ID starting with 1146f7cb3cf170ef3a171d143b08788ad7aa44257c4ef3bfc2eac1b7ce391a70 not found: ID does not exist" Jul 16 22:33:07.017845 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:07.017752 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-openshift-service-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:33:07.017845 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:07.017786 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed-proxy-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:33:07.247161 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:07.247127 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w"] Jul 16 22:33:07.253281 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:07.253255 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-76a97-8669877574-rbn4w"] Jul 16 22:33:07.264037 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:33:07.261778 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" path="/var/lib/kubelet/pods/e2e1c182-96b5-478f-a3a5-b5bef18bc6ed/volumes" Jul 16 22:40:06.916610 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:06.916574 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc"] Jul 16 22:40:06.919143 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:06.916768 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" podUID="5b417db4-3fb1-4246-b106-5113a3334c64" containerName="switch-graph-93462" containerID="cri-o://a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01" gracePeriod=30 Jul 16 22:40:09.878462 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:09.878423 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" podUID="5b417db4-3fb1-4246-b106-5113a3334c64" containerName="switch-graph-93462" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:40:14.878825 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:14.878776 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" podUID="5b417db4-3fb1-4246-b106-5113a3334c64" containerName="switch-graph-93462" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:40:19.878622 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:19.878578 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" podUID="5b417db4-3fb1-4246-b106-5113a3334c64" containerName="switch-graph-93462" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:40:19.879028 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:19.878683 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:40:23.851555 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:23.851523 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:24.732503 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:24.732474 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:24.879071 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:24.879035 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" podUID="5b417db4-3fb1-4246-b106-5113a3334c64" containerName="switch-graph-93462" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:40:25.628640 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:25.628608 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:26.534051 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:26.534024 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:27.364409 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:27.364375 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:28.247119 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:28.247087 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:29.184756 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:29.184717 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:29.878638 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:29.878593 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" podUID="5b417db4-3fb1-4246-b106-5113a3334c64" containerName="switch-graph-93462" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:40:30.182491 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:30.182457 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:31.216409 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:31.216378 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:32.238114 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:32.238071 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:33.185011 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:33.184982 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:34.156132 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:34.156106 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:34.878293 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:34.878258 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" podUID="5b417db4-3fb1-4246-b106-5113a3334c64" containerName="switch-graph-93462" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 22:40:35.094701 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:35.094652 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:36.034026 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:36.033997 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-93462-6584468b49-vtflc_5b417db4-3fb1-4246-b106-5113a3334c64/switch-graph-93462/0.log" Jul 16 22:40:37.063806 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.063783 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:40:37.145399 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.145371 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b417db4-3fb1-4246-b106-5113a3334c64-openshift-service-ca-bundle\") pod \"5b417db4-3fb1-4246-b106-5113a3334c64\" (UID: \"5b417db4-3fb1-4246-b106-5113a3334c64\") " Jul 16 22:40:37.145559 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.145415 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b417db4-3fb1-4246-b106-5113a3334c64-proxy-tls\") pod \"5b417db4-3fb1-4246-b106-5113a3334c64\" (UID: \"5b417db4-3fb1-4246-b106-5113a3334c64\") " Jul 16 22:40:37.145808 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.145781 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b417db4-3fb1-4246-b106-5113a3334c64-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5b417db4-3fb1-4246-b106-5113a3334c64" (UID: "5b417db4-3fb1-4246-b106-5113a3334c64"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:40:37.147559 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.147537 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b417db4-3fb1-4246-b106-5113a3334c64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b417db4-3fb1-4246-b106-5113a3334c64" (UID: "5b417db4-3fb1-4246-b106-5113a3334c64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:40:37.246158 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.246127 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b417db4-3fb1-4246-b106-5113a3334c64-proxy-tls\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:40:37.246158 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.246157 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b417db4-3fb1-4246-b106-5113a3334c64-openshift-service-ca-bundle\") on node \"ip-10-0-131-231.ec2.internal\" DevicePath \"\"" Jul 16 22:40:37.555277 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.555192 2558 generic.go:356] "Generic (PLEG): container finished" podID="5b417db4-3fb1-4246-b106-5113a3334c64" containerID="a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01" exitCode=0 Jul 16 22:40:37.555277 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.555251 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" Jul 16 22:40:37.555451 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.555273 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" event={"ID":"5b417db4-3fb1-4246-b106-5113a3334c64","Type":"ContainerDied","Data":"a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01"} Jul 16 22:40:37.555451 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.555330 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc" event={"ID":"5b417db4-3fb1-4246-b106-5113a3334c64","Type":"ContainerDied","Data":"7e2c64d80809d262a42aa23006dc1b585e2fb057e058c8944568ab9d143bc259"} Jul 16 22:40:37.555451 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.555351 2558 scope.go:117] "RemoveContainer" containerID="a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01" Jul 16 22:40:37.563346 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.563328 2558 scope.go:117] "RemoveContainer" containerID="a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01" Jul 16 22:40:37.563625 ip-10-0-131-231 kubenswrapper[2558]: E0716 22:40:37.563607 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01\": container with ID starting with a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01 not found: ID does not exist" containerID="a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01" Jul 16 22:40:37.563669 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.563635 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01"} err="failed to get container status \"a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01\": rpc error: code = NotFound desc = could not find container \"a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01\": container with ID starting with a93a66b06c64d3a9f7434e47d63e1d92d396f9dcceb3dbea85c1ca12f3081d01 not found: ID does not exist" Jul 16 22:40:37.581496 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.581471 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc"] Jul 16 22:40:37.589214 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:37.589191 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-93462-6584468b49-vtflc"] Jul 16 22:40:39.261441 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:39.261409 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b417db4-3fb1-4246-b106-5113a3334c64" path="/var/lib/kubelet/pods/5b417db4-3fb1-4246-b106-5113a3334c64/volumes" Jul 16 22:40:40.438748 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.438712 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jnrjl/must-gather-7pq5f"] Jul 16 22:40:40.439172 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.439002 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" containerName="splitter-graph-76a97" Jul 16 22:40:40.439172 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.439013 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" containerName="splitter-graph-76a97" Jul 16 22:40:40.439172 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.439030 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b417db4-3fb1-4246-b106-5113a3334c64" containerName="switch-graph-93462" Jul 16 22:40:40.439172 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.439036 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b417db4-3fb1-4246-b106-5113a3334c64" containerName="switch-graph-93462" Jul 16 22:40:40.439172 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.439087 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e1c182-96b5-478f-a3a5-b5bef18bc6ed" containerName="splitter-graph-76a97" Jul 16 22:40:40.439172 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.439097 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b417db4-3fb1-4246-b106-5113a3334c64" containerName="switch-graph-93462" Jul 16 22:40:40.442187 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.442170 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jnrjl/must-gather-7pq5f" Jul 16 22:40:40.451645 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.451628 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jnrjl\"/\"default-dockercfg-gv8jf\"" Jul 16 22:40:40.452504 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.452465 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jnrjl\"/\"openshift-service-ca.crt\"" Jul 16 22:40:40.452629 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.452612 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jnrjl\"/\"kube-root-ca.crt\"" Jul 16 22:40:40.470774 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.470743 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjq6\" (UniqueName: \"kubernetes.io/projected/2320d7e1-8a4c-421f-8b84-d64204a144e1-kube-api-access-rmjq6\") pod \"must-gather-7pq5f\" (UID: \"2320d7e1-8a4c-421f-8b84-d64204a144e1\") " pod="openshift-must-gather-jnrjl/must-gather-7pq5f" Jul 16 22:40:40.470932 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.470808 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2320d7e1-8a4c-421f-8b84-d64204a144e1-must-gather-output\") pod \"must-gather-7pq5f\" (UID: \"2320d7e1-8a4c-421f-8b84-d64204a144e1\") " pod="openshift-must-gather-jnrjl/must-gather-7pq5f" Jul 16 22:40:40.479819 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.479785 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jnrjl/must-gather-7pq5f"] Jul 16 22:40:40.571715 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.571682 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjq6\" (UniqueName: \"kubernetes.io/projected/2320d7e1-8a4c-421f-8b84-d64204a144e1-kube-api-access-rmjq6\") pod \"must-gather-7pq5f\" (UID: \"2320d7e1-8a4c-421f-8b84-d64204a144e1\") " pod="openshift-must-gather-jnrjl/must-gather-7pq5f" Jul 16 22:40:40.571909 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.571733 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2320d7e1-8a4c-421f-8b84-d64204a144e1-must-gather-output\") pod \"must-gather-7pq5f\" (UID: \"2320d7e1-8a4c-421f-8b84-d64204a144e1\") " pod="openshift-must-gather-jnrjl/must-gather-7pq5f" Jul 16 22:40:40.572067 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.572047 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2320d7e1-8a4c-421f-8b84-d64204a144e1-must-gather-output\") pod \"must-gather-7pq5f\" (UID: \"2320d7e1-8a4c-421f-8b84-d64204a144e1\") " pod="openshift-must-gather-jnrjl/must-gather-7pq5f" Jul 16 22:40:40.592201 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.592178 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjq6\" (UniqueName: \"kubernetes.io/projected/2320d7e1-8a4c-421f-8b84-d64204a144e1-kube-api-access-rmjq6\") pod \"must-gather-7pq5f\" (UID: \"2320d7e1-8a4c-421f-8b84-d64204a144e1\") " pod="openshift-must-gather-jnrjl/must-gather-7pq5f" Jul 16 22:40:40.767370 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.767276 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jnrjl/must-gather-7pq5f" Jul 16 22:40:40.929146 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.929115 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jnrjl/must-gather-7pq5f"] Jul 16 22:40:40.932810 ip-10-0-131-231 kubenswrapper[2558]: W0716 22:40:40.932785 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2320d7e1_8a4c_421f_8b84_d64204a144e1.slice/crio-a7928cb752a9018199ae5e060d454ea743e2635a358dacb5511dbac0fb4a7c4c WatchSource:0}: Error finding container a7928cb752a9018199ae5e060d454ea743e2635a358dacb5511dbac0fb4a7c4c: Status 404 returned error can't find the container with id a7928cb752a9018199ae5e060d454ea743e2635a358dacb5511dbac0fb4a7c4c Jul 16 22:40:40.934847 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:40.934833 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 22:40:41.574817 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:41.574786 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jnrjl/must-gather-7pq5f" event={"ID":"2320d7e1-8a4c-421f-8b84-d64204a144e1","Type":"ContainerStarted","Data":"a7928cb752a9018199ae5e060d454ea743e2635a358dacb5511dbac0fb4a7c4c"} Jul 16 22:40:42.581923 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:42.581881 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jnrjl/must-gather-7pq5f" event={"ID":"2320d7e1-8a4c-421f-8b84-d64204a144e1","Type":"ContainerStarted","Data":"0abf05ad7f0ae43e7e123a82489bbdb211310b5ef86c95f64645edaa4023f04e"} Jul 16 22:40:42.582400 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:42.581938 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jnrjl/must-gather-7pq5f" event={"ID":"2320d7e1-8a4c-421f-8b84-d64204a144e1","Type":"ContainerStarted","Data":"9e46eed5bb21ebb3c69a552086a09326ed9087ec733b3ff23490060f95291d60"} Jul 16 22:40:42.606552 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:42.606500 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jnrjl/must-gather-7pq5f" podStartSLOduration=1.514791408 podStartE2EDuration="2.606484266s" totalImagesPullingTime="1.091692858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:42.605456253 +0000 UTC m=+3043.865366677" watchObservedRunningTime="2026-07-16 22:40:42.606484266 +0000 UTC m=+3043.866394729" Jul 16 22:40:43.793377 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:43.793350 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jkdtg_9c1ed479-ddc7-4099-b273-ddd8c98534b4/global-pull-secret-syncer/0.log" Jul 16 22:40:43.929465 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:43.929437 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dmp8k_079499fc-0421-434a-8a61-d0d075b89be1/konnectivity-agent/0.log" Jul 16 22:40:44.062068 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:44.061988 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-231.ec2.internal_f66b8fb68e2cffebe18d00c9a2172a9c/haproxy/0.log" Jul 16 22:40:48.710723 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:48.710695 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wwmt_5aead329-3038-4b94-b6ae-6879d73fb7b7/node-exporter/0.log" Jul 16 22:40:48.746078 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:48.745994 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wwmt_5aead329-3038-4b94-b6ae-6879d73fb7b7/kube-rbac-proxy/0.log" Jul 16 22:40:48.794051 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:48.793910 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wwmt_5aead329-3038-4b94-b6ae-6879d73fb7b7/init-textfile/0.log" Jul 16 22:40:48.970288 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:48.970237 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-n5l8q_6a745b7c-fdb1-401b-8ebb-5770258e249e/kube-rbac-proxy-main/0.log" Jul 16 22:40:49.016103 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:49.016071 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-n5l8q_6a745b7c-fdb1-401b-8ebb-5770258e249e/kube-rbac-proxy-self/0.log" Jul 16 22:40:49.060439 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:49.060409 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-n5l8q_6a745b7c-fdb1-401b-8ebb-5770258e249e/openshift-state-metrics/0.log" Jul 16 22:40:49.120735 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:49.120702 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aeebd685-e312-4848-adbb-2b74fccf2079/prometheus/0.log" Jul 16 22:40:49.176076 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:49.176048 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aeebd685-e312-4848-adbb-2b74fccf2079/config-reloader/0.log" Jul 16 22:40:49.254194 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:49.254097 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aeebd685-e312-4848-adbb-2b74fccf2079/thanos-sidecar/0.log" Jul 16 22:40:49.330773 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:49.330736 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aeebd685-e312-4848-adbb-2b74fccf2079/kube-rbac-proxy-web/0.log" Jul 16 22:40:49.430252 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:49.430216 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aeebd685-e312-4848-adbb-2b74fccf2079/kube-rbac-proxy/0.log" Jul 16 22:40:49.483875 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:49.483820 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aeebd685-e312-4848-adbb-2b74fccf2079/kube-rbac-proxy-thanos/0.log" Jul 16 22:40:49.521160 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:49.521044 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_aeebd685-e312-4848-adbb-2b74fccf2079/init-config-reloader/0.log" Jul 16 22:40:51.949601 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:51.949555 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f"] Jul 16 22:40:51.955079 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:51.955053 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:51.968563 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:51.968533 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f"] Jul 16 22:40:51.968909 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:51.968879 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gs7h\" (UniqueName: \"kubernetes.io/projected/8a34d055-e35a-44f6-ab96-c2846332f756-kube-api-access-5gs7h\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:51.969051 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:51.968947 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-lib-modules\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:51.969051 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:51.968986 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-podres\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:51.969051 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:51.969048 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-sys\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:51.969245 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:51.969077 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-proc\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.069795 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.069754 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gs7h\" (UniqueName: \"kubernetes.io/projected/8a34d055-e35a-44f6-ab96-c2846332f756-kube-api-access-5gs7h\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.070119 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.070097 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-lib-modules\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.070287 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.070271 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-podres\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.070474 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.070441 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-podres\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.070577 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.070468 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-lib-modules\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.070577 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.070496 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-sys\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.070577 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.070447 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-sys\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.070577 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.070560 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-proc\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.070741 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.070638 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a34d055-e35a-44f6-ab96-c2846332f756-proc\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.080161 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.080128 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gs7h\" (UniqueName: \"kubernetes.io/projected/8a34d055-e35a-44f6-ab96-c2846332f756-kube-api-access-5gs7h\") pod \"perf-node-gather-daemonset-2gz9f\" (UID: \"8a34d055-e35a-44f6-ab96-c2846332f756\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.267964 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.267877 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.435094 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.434400 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f"] Jul 16 22:40:52.639196 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.639152 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" event={"ID":"8a34d055-e35a-44f6-ab96-c2846332f756","Type":"ContainerStarted","Data":"f75299f159d8fbeed0ad7a564efd22b64c55746ca0b27ea8fd94c5b6b8e73f49"} Jul 16 22:40:52.639463 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.639222 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" event={"ID":"8a34d055-e35a-44f6-ab96-c2846332f756","Type":"ContainerStarted","Data":"238eb83acf7bf7001570fa649f558df50d121494008c7b7166562c5c19db868d"} Jul 16 22:40:52.639463 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.639269 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:40:52.659498 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:52.659440 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" podStartSLOduration=1.659420652 podStartE2EDuration="1.659420652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:52.658895401 +0000 UTC m=+3053.918805856" watchObservedRunningTime="2026-07-16 22:40:52.659420652 +0000 UTC m=+3053.919331077" Jul 16 22:40:54.083411 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:54.083374 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xgq9j_3c0c065d-3277-4995-beca-874814b115fa/dns/0.log" Jul 16 22:40:54.111643 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:54.111613 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xgq9j_3c0c065d-3277-4995-beca-874814b115fa/kube-rbac-proxy/0.log" Jul 16 22:40:54.141238 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:54.141201 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b2wz8_c29a9f8e-4098-45a1-8c53-84c0fc20f0c1/dns-node-resolver/0.log" Jul 16 22:40:54.737154 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:54.737127 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-964956bf6-6tdrb_78e10644-eb84-4ed1-9100-439190cd5bec/registry/0.log" Jul 16 22:40:54.772685 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:54.772658 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8stlx_3347b12c-12e0-4f72-a9eb-9480db6fd042/node-ca/0.log" Jul 16 22:40:55.718242 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:55.718219 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c99d8d4d4-4trjg_5db1862a-2a05-4cf2-bc16-05f4d1188f88/router/0.log" Jul 16 22:40:56.174121 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:56.174095 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mz7fk_cad724ee-e0b4-4519-9f88-52e577024ced/serve-healthcheck-canary/0.log" Jul 16 22:40:56.802618 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:56.802571 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89sfs_845e715f-d30d-4a40-b41c-edb7671ae0a2/kube-rbac-proxy/0.log" Jul 16 22:40:56.856158 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:56.856126 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89sfs_845e715f-d30d-4a40-b41c-edb7671ae0a2/exporter/0.log" Jul 16 22:40:56.914430 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:56.914400 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89sfs_845e715f-d30d-4a40-b41c-edb7671ae0a2/extractor/0.log" Jul 16 22:40:58.655429 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:40:58.655403 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-2gz9f" Jul 16 22:41:04.765246 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:04.765216 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-brx68_bd52e79a-6250-458a-8547-6c4aa794f7d2/migrator/0.log" Jul 16 22:41:04.805670 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:04.805638 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-brx68_bd52e79a-6250-458a-8547-6c4aa794f7d2/graceful-termination/0.log" Jul 16 22:41:05.210693 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:05.210646 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-s9jq4_5cd0ba43-b852-4efc-bfb9-0303473bb89c/kube-storage-version-migrator-operator/1.log" Jul 16 22:41:05.212271 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:05.212245 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-s9jq4_5cd0ba43-b852-4efc-bfb9-0303473bb89c/kube-storage-version-migrator-operator/0.log" Jul 16 22:41:06.260469 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:06.260440 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5x7sg_cd481684-0c85-4cbe-ab71-3e6130ab1131/kube-multus-additional-cni-plugins/0.log" Jul 16 22:41:06.286934 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:06.286908 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5x7sg_cd481684-0c85-4cbe-ab71-3e6130ab1131/egress-router-binary-copy/0.log" Jul 16 22:41:06.314335 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:06.314311 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5x7sg_cd481684-0c85-4cbe-ab71-3e6130ab1131/cni-plugins/0.log" Jul 16 22:41:06.342169 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:06.342146 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5x7sg_cd481684-0c85-4cbe-ab71-3e6130ab1131/bond-cni-plugin/0.log" Jul 16 22:41:06.370319 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:06.370289 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5x7sg_cd481684-0c85-4cbe-ab71-3e6130ab1131/routeoverride-cni/0.log" Jul 16 22:41:06.404629 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:06.404596 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5x7sg_cd481684-0c85-4cbe-ab71-3e6130ab1131/whereabouts-cni-bincopy/0.log" Jul 16 22:41:06.431863 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:06.431840 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5x7sg_cd481684-0c85-4cbe-ab71-3e6130ab1131/whereabouts-cni/0.log" Jul 16 22:41:06.912583 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:06.912559 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bzncd_028411f0-adf7-453e-bb59-1b27079457e4/kube-multus/0.log" Jul 16 22:41:07.116493 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:07.116466 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wkshg_7e676b3e-e7cf-41d1-ac8c-e2177080ea66/network-metrics-daemon/0.log" Jul 16 22:41:07.146136 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:07.146110 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wkshg_7e676b3e-e7cf-41d1-ac8c-e2177080ea66/kube-rbac-proxy/0.log" Jul 16 22:41:08.308777 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:08.308750 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ttlc_a7fe9137-5027-40c2-aebf-9c0a64d1fe37/ovn-controller/0.log" Jul 16 22:41:08.336834 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:08.336808 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ttlc_a7fe9137-5027-40c2-aebf-9c0a64d1fe37/ovn-acl-logging/0.log" Jul 16 22:41:08.370767 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:08.370732 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ttlc_a7fe9137-5027-40c2-aebf-9c0a64d1fe37/kube-rbac-proxy-node/0.log" Jul 16 22:41:08.403530 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:08.403503 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ttlc_a7fe9137-5027-40c2-aebf-9c0a64d1fe37/kube-rbac-proxy-ovn-metrics/0.log" Jul 16 22:41:08.429933 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:08.429909 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ttlc_a7fe9137-5027-40c2-aebf-9c0a64d1fe37/northd/0.log" Jul 16 22:41:08.460094 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:08.460071 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ttlc_a7fe9137-5027-40c2-aebf-9c0a64d1fe37/nbdb/0.log" Jul 16 22:41:08.492012 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:08.491985 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ttlc_a7fe9137-5027-40c2-aebf-9c0a64d1fe37/sbdb/0.log" Jul 16 22:41:08.542567 ip-10-0-131-231 kubenswrapper[2558]: I0716 22:41:08.542536 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7ttlc_a7fe9137-5027-40c2-aebf-9c0a64d1fe37/ovnkube-controller/0.log"