Sep 16 18:38:29.523045 ip-10-0-131-137 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:38:29.523052 ip-10-0-131-137 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:38:29.523061 ip-10-0-131-137 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:38:29.523388 ip-10-0-131-137 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:38:38.841547 ip-10-0-131-137 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 16 18:38:38.841559 ip-10-0-131-137 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 45ea0f63b69d4cd882b7cc69b5d0000c --
Sep 16 18:40:26.883060 ip-10-0-131-137 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:40:27.428487 ip-10-0-131-137 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:27.428487 ip-10-0-131-137 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:40:27.428487 ip-10-0-131-137 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:27.428487 ip-10-0-131-137 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:40:27.428487 ip-10-0-131-137 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:40:27.431697 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.431607    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:40:27.437408 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437392    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:27.437408 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437408    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437411    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437414    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437417    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437420    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437423    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437426    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437429    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437431    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437435    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437438    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437441    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437443    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437446    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437449    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437460    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437463    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437465    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437468    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:27.437475 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437471    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437473    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437476    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437478    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437481    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437484    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437487    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437490    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437492    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437494    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437497    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437500    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437503    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437505    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437509    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437513    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437516    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437519    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437522    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:27.437953 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437524    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437527    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437530    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437532    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437535    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437540    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437542    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437545    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437547    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437550    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437553    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437556    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437558    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437561    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437563    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437566    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437569    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437572    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437574    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437577    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:27.438414 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437579    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437582    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437584    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437587    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437589    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437592    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437595    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437597    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437599    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437602    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437605    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437607    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437610    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437612    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437614    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437617    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437619    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437622    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437625    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437628    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:27.438939 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437630    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437633    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437635    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437638    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437642    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437645    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437648    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437651    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437653    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437656    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437658    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437661    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437663    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437667    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437671    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437673    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437677    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437680    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437683    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:27.439423 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.437686    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438450    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438457    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438461    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438463    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438466    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438469    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438471    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438474    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438477    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438479    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438482    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438485    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438487    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438490    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438493    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438495    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438498    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438501    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:27.439917 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438504    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438507    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438509    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438512    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438515    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438517    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438520    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438522    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438525    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438527    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438530    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438532    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438534    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438537    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438539    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438542    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438545    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438547    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438550    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438553    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:27.440647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438556    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438558    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438561    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438564    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438566    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438569    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438571    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438573    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438576    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438578    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438581    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438584    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438586    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438588    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438591    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438593    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438596    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438598    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438601    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438603    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:27.441481 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438606    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438608    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438611    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438613    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438616    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438619    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438621    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438623    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438626    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438628    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438631    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438634    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438637    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438640    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438644    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438646    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438649    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438651    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438654    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438656    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:27.442107 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438659    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438661    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438663    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438666    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438669    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438671    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438674    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438676    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438679    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438681    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438685    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438689    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438692    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438695    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438698    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438702    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438705    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438708    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438712    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:27.442593 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438715    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.438718    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439123    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439129    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439133    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439136    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439139    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439142    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439145    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439147    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439150    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439153    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439155    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439158    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439160    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439163    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439165    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439168    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439171    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439173    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:27.443078 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439176    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439179    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439181    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439184    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439186    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439189    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439192    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439194    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439196    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439199    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439202    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439204    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439207    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439209    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439212    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439214    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439217    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439221    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439224    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:27.443566 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439227    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439230    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439232    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439235    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439237    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439240    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439243    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439245    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439248    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439250    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439253    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439263    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439266    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439268    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439271    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439273    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439276    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439278    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439281    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439283    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:27.444060 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439285    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439288    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439290    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439293    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439297    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439300    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439302    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439305    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439307    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439310    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439312    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439315    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439317    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439320    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439323    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439325    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439328    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439330    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439333    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:27.444572 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439335    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439340    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439343    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439346    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439350    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439360    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439363    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439366    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439368    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439371    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439373    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439376    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439379    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439381    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439383    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439386    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439390    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439393    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439396    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:27.445155 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439398    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439401    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439403    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.439406    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439483    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439517    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439527    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439531    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439535    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439538    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439543    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439547    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439550    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439553    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439557    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439560    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439563    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439566    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439569    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439581    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439584    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439587    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439593    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439596    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:40:27.445617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439599    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439602    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439606    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439609    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439621    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439624    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439628    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439631    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439634    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439637    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439640    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439643    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439647    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439650    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439652    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439655    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439658    2583 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439661    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439668    2583 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439671    2583 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439673    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439676    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439684    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439688    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:40:27.446209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439691    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439694    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439697    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439700    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439708    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439711    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439714    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439717    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439719    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439722    2583 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439726    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439729    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439732    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439735    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439753    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439756    2583 flags.go:64] FLAG: --help="false"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439759    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-131-137.ec2.internal"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439762    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439765    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439767    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439771    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439774    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439777    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:40:27.446819 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439780    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439783    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439786    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439789    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439792    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439794    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439797    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439800    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439802    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439805    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439808    2583 flags.go:64] FLAG: --lock-file=""
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439810    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439813    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439816    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439827    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439830    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439833    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439835    2583 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439838    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439841    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439847    2583 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439850    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439855    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439858    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439862    2583 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:40:27.447424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439865    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439868    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439870    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439873    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439876    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439879    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439882    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439890    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439893    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439896    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439899    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439901    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439907    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439909    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439912    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439915    2583 flags.go:64] FLAG: --port="10250"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439918    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439921    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07254ebd69f71c906"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439924    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439927    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439930    2583 flags.go:64] FLAG: --register-node="true"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439933    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.439982    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441098    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:40:27.448102 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441105    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441108    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441113    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441116    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441120    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441123    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441126    2583 flags.go:64] FLAG: --runonce="false"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441129    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441136    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441140    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441143    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441146    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441149    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441153    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441156    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441159    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441165    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441168    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441171    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441174    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441179    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441183    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441190    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441197    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441200    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:40:27.448680 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441214    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441217    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441220    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441223    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441226    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441229    2583 flags.go:64] FLAG: --v="2"
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441262    2583 flags.go:64] FLAG: --version="false"
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441266    2583 flags.go:64] FLAG: --vmodule=""
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441270    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.441274    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441643    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441648    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441651    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441658    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441661    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441663    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441666    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441669    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441673    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441680    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441683    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441686    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:27.449344 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441689    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441692    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441694    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441697    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441700    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441704    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441706    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441709    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441712    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441715    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441720    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441723    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441726    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441729    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441731    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441734    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441752    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441811    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441830    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:27.449895 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441834    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441944    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441947    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441950    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441955    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441958    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441961    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441963    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441966    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441969    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441971    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441974    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441976    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441979    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441982    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441985    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441987    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441989    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.441994    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442000    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:27.450362 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442003    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442005    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442008    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442010    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442013    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442016    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442018    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442021    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442023    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442026    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442029    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442031    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442034    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442037    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442041    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442044    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442048    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442050    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442053    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442055    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:27.450861 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442058    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442060    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442065    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442068    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442071    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442074    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442077    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442080    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442083    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442087    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442091    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442094    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442097    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442099    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442102    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442105    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442108    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442110    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442113    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:27.451353 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442116    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:27.451892 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442119    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:27.451892 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442121    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:27.451892 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442124    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:27.451892 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442126    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:27.451892 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442129    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:27.451892 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442132    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:27.451892 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442134    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:27.451892 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.442137    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:27.451892 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.442839    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:27.451892 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.451126    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:40:27.451892 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.451142    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.452011    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.452019    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452080    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452085    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452089    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452092    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452095    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452098    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452100    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452111    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452114    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452117    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452119    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452122    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452125    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452128    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452130    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452133    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452136    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:27.452182 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452139    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452141    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452144    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452146    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452149    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452151    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452154    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452157    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452159    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452162    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452164    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452167    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452169    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452171    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452175    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452177    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452180    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452183    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452186    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452188    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:27.452659 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452191    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452194    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452196    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452205    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452208    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452210    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452212    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452215    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452218    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452220    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452223    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452225    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452228    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452230    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452234    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452238    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452241    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452244    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452247    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452250    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:27.453150 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452252    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452255    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452259    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452262    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452265    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452269    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452272    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452277    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452280    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452283    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452286    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452289    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452292    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452295    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452298    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452300    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452304    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452306    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452309    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:27.453630 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452311    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452314    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452316    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452319    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452322    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452324    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452327    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452329    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452331    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452334    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452336    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452339    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452341    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452344    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452346    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452349    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452351    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452354    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452356    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:27.454119 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452360    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452362    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452365    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452368    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.452373    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452495    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452499    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452502    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452505    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452508    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452511    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452515    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452518    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452520    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452523    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:40:27.454581 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452526    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452528    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452531    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452534    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452536    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452539    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452541    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452544    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452546    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452549    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452551    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452554    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452556    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452559    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452561    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452565    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452569    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452573    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452576    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452579    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:40:27.454965 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452581    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452584    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452587    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452589    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452592    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452595    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452597    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452600    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452602    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452606    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452609    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452611    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452614    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452616    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452619    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452621    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452624    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452627    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452629    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452632    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:40:27.455454 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452634    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452637    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452639    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452642    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452644    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452647    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452649    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452652    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452654    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452657    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452659    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452662    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452665    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452667    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452670    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452673    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452675    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452678    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452680    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452683    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:40:27.455959 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452686    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452688    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452691    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452694    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452696    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452699    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452701    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452703    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452706    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452708    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452711    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452713    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452717    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452720    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452723    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452726    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452728    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452730    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452733    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:40:27.456457 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452750    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452754    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452757    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452761    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452764    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452767    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452769    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452772    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452775    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:27.452778    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.452784    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.452899    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:40:27.457059 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.456278    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:40:27.457387 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.457375    2583 server.go:1019] "Starting client certificate rotation"
Sep 16 18:40:27.457497 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.457482    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:27.457529 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.457514    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:40:27.460191 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.460177    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:27.464163 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.464149    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:40:27.478173 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.478153    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:40:27.483484 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.483468    2583 log.go:25] "Validated CRI v1 image API"
Sep 16 18:40:27.484373 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.484354    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:40:27.486925 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.486907    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7b4c0d9a-4a5e-4355-9444-2b30801d7b47:/dev/nvme0n1p4 d271522f-916d-4f22-972d-347bdb22d693:/dev/nvme0n1p3]
Sep 16 18:40:27.486978 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.486926    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:40:27.492253 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.492238    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:27.492441 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.492325    2583 manager.go:216] Machine: {Timestamp:2026-09-16 18:40:27.490926714 +0000 UTC m=+0.373749787 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105178 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dcbe958e0edd9c6ae45776ab5af02 SystemUUID:ec2dcbe9-58e0-edd9-c6ae-45776ab5af02 BootID:45ea0f63-b69d-4cd8-82b7-cc69b5d0000c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f9:c0:46:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f9:c0:46:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:01:24:41:e6:9c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:40:27.492441 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.492440    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:40:27.492539 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.492515    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:40:27.496096 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.496069    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:40:27.496227 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.496099    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-137.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:40:27.496952 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.496942    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:40:27.496989 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.496956    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:40:27.496989 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.496964    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:27.497043 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.496994    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:40:27.498177 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.498168    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:40:27.502779 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.502767    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:27.502901 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.502891    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:40:27.502930 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.502920    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:40:27.502956 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.502931    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:40:27.502956 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.502944    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:40:27.502956 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.502954    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:40:27.503041 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.502963    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:40:27.503278 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.503266    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:27.503337 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.503290    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:40:27.505540 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.505507    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:40:27.507712 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.507698    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:40:27.507769 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.507729    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:40:27.509811 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509800    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:40:27.509869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509816    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:40:27.509869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509823    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:40:27.509869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509829    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:40:27.509869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509837    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:40:27.509869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509842    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:40:27.509869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509855    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:40:27.509869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509861    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:40:27.509869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509868    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:40:27.509869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509873    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:40:27.510136 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509882    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:40:27.510136 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.509887    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:40:27.510852 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.510843    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:40:27.510852 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.510852    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:40:27.514732 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.514718    2583 server.go:1268] "Started kubelet"
Sep 16 18:40:27.514859 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.514814    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:40:27.515203 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.515186    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-137.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:40:27.515459 ip-10-0-131-137 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:40:27.515621 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.515534    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:40:27.515621 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.515597    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:40:27.516423 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.515672    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:40:27.516423 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.516304    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:40:27.518582 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.518550    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:40:27.519270 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.519255    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:40:27.522112 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.522091    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:40:27.524408 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.524385    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:27.525048 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.525029    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:40:27.525180 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.525130    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:27.525889 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.525711    2583 factory.go:55] Registering systemd factory
Sep 16 18:40:27.525965 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.525893    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:40:27.525965 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.525714    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:40:27.525965 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.524708    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-137.ec2.internal.18d5e130ffb9d30b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-137.ec2.internal,UID:ip-10-0-131-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-137.ec2.internal,},FirstTimestamp:2026-09-16 18:40:27.514688267 +0000 UTC m=+0.397511340,LastTimestamp:2026-09-16 18:40:27.514688267 +0000 UTC m=+0.397511340,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-137.ec2.internal,}"
Sep 16 18:40:27.525965 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.525716    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:40:27.525965 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.525952    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:40:27.526166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.526027    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:40:27.526166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.526036    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:40:27.526166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.526100    2583 factory.go:153] Registering CRI-O factory
Sep 16 18:40:27.526166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.526114    2583 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:40:27.526166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.526167    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:40:27.526356 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.526180    2583 factory.go:103] Registering Raw factory
Sep 16 18:40:27.526356 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.526190    2583 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:40:27.526541 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.526529    2583 manager.go:349] Starting recovery of all containers
Sep 16 18:40:27.532945 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.532918    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:40:27.533032 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.532957    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:40:27.536631 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.536613    2583 manager.go:354] Recovery completed
Sep 16 18:40:27.538304 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.538275    2583 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 16 18:40:27.542297 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.542281    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:27.544083 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.544060    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:27.544178 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.544096    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:27.544178 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.544105    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:27.544677 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.544665    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:40:27.544734 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.544677    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:27.544734 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.544677    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:40:27.544822 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.544765    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:40:27.545939 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.545851    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-137.ec2.internal.18d5e131017a58d2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-137.ec2.internal,UID:ip-10-0-131-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-137.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-137.ec2.internal,},FirstTimestamp:2026-09-16 18:40:27.544082642 +0000 UTC m=+0.426905713,LastTimestamp:2026-09-16 18:40:27.544082642 +0000 UTC m=+0.426905713,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-137.ec2.internal,}"
Sep 16 18:40:27.546142 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.546121    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:27.546223 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.546171    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:27.546223 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.546189    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:27.546994 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.546979    2583 policy_none.go:49] "None policy: Start"
Sep 16 18:40:27.546994 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.546996    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:40:27.547076 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.547006    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:40:27.548995 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.548985    2583 policy_none.go:47] "Start"
Sep 16 18:40:27.556863 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.556795    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-137.ec2.internal.18d5e131017a9d1e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-137.ec2.internal,UID:ip-10-0-131-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-137.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-137.ec2.internal,},FirstTimestamp:2026-09-16 18:40:27.544100126 +0000 UTC m=+0.426923199,LastTimestamp:2026-09-16 18:40:27.544100126 +0000 UTC m=+0.426923199,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-137.ec2.internal,}"
Sep 16 18:40:27.567298 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.567210    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-137.ec2.internal.18d5e131017ac028  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-137.ec2.internal,UID:ip-10-0-131-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-137.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-137.ec2.internal,},FirstTimestamp:2026-09-16 18:40:27.544109096 +0000 UTC m=+0.426932168,LastTimestamp:2026-09-16 18:40:27.544109096 +0000 UTC m=+0.426932168,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-137.ec2.internal,}"
Sep 16 18:40:27.576818 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.576755    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-137.ec2.internal.18d5e131017a58d2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-137.ec2.internal.18d5e131017a58d2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-137.ec2.internal,UID:ip-10-0-131-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-137.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-137.ec2.internal,},FirstTimestamp:2026-09-16 18:40:27.544082642 +0000 UTC m=+0.426905713,LastTimestamp:2026-09-16 18:40:27.546140997 +0000 UTC m=+0.428964072,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-137.ec2.internal,}"
Sep 16 18:40:27.583310 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.583296    2583 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.583335    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.583348    2583 server.go:87] "Starting device plugin registration server"
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.583620    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.583633    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.583724    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.583853    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.583864    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.584264    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.584311    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.589002    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-137.ec2.internal.18d5e131017a9d1e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-137.ec2.internal.18d5e131017a9d1e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-137.ec2.internal,UID:ip-10-0-131-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-137.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-137.ec2.internal,},FirstTimestamp:2026-09-16 18:40:27.544100126 +0000 UTC m=+0.426923199,LastTimestamp:2026-09-16 18:40:27.546179634 +0000 UTC m=+0.429002897,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-137.ec2.internal,}"
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.597850    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-137.ec2.internal.18d5e131017ac028\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-137.ec2.internal.18d5e131017ac028  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-137.ec2.internal,UID:ip-10-0-131-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-137.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-137.ec2.internal,},FirstTimestamp:2026-09-16 18:40:27.544109096 +0000 UTC m=+0.426932168,LastTimestamp:2026-09-16 18:40:27.546194588 +0000 UTC m=+0.429017660,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-137.ec2.internal,}"
Sep 16 18:40:27.606334 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.605154    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-137.ec2.internal.18d5e13104480e7d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-137.ec2.internal,UID:ip-10-0-131-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-131-137.ec2.internal,},FirstTimestamp:2026-09-16 18:40:27.591118461 +0000 UTC m=+0.473941520,LastTimestamp:2026-09-16 18:40:27.591118461 +0000 UTC m=+0.473941520,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-137.ec2.internal,}"
Sep 16 18:40:27.651057 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.651027    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:40:27.652319 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.652299    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:40:27.652403 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.652363    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:40:27.652403 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.652391    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:40:27.652477 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.652436    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:40:27.655337 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.655312    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjg27"
Sep 16 18:40:27.664711 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.664691    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 16 18:40:27.664897 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.664884    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjg27"
Sep 16 18:40:27.683859 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.683799    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:27.684593 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.684577    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:27.684663 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.684611    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:27.684663 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.684626    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:27.684663 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.684654    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-137.ec2.internal"
Sep 16 18:40:27.702174 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.702157    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-137.ec2.internal"
Sep 16 18:40:27.702260 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.702177    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-137.ec2.internal\": node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:27.752790 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.752763    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-137.ec2.internal"]
Sep 16 18:40:27.752918 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.752833    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:27.754312 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.754298    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:27.754405 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.754330    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:27.754405 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.754344    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:27.756637 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.756622    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:27.756759 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.756726    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:27.756811 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.756773    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:27.757356 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.757341    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:27.757356 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.757349    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:27.757455 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.757369    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:27.757455 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.757368    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:27.757455 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.757380    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:27.757455 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.757388    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:27.760157 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.760143    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:27.760225 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.760168    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:27.760842 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.760825    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:27.760922 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.760853    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:27.760922 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.760863    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:27.781249 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.781231    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-137.ec2.internal\" not found" node="ip-10-0-131-137.ec2.internal"
Sep 16 18:40:27.785492 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.785473    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-137.ec2.internal\" not found" node="ip-10-0-131-137.ec2.internal"
Sep 16 18:40:27.822535 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.822513    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:27.923053 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:27.923028    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:27.927302 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.927284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76002f047f028799fe97fbc79e29015d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal\" (UID: \"76002f047f028799fe97fbc79e29015d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:27.927356 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.927310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae92e6bf8b06378903825aae1b8252ca-config\") pod \"kube-apiserver-proxy-ip-10-0-131-137.ec2.internal\" (UID: \"ae92e6bf8b06378903825aae1b8252ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:27.927356 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:27.927327    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76002f047f028799fe97fbc79e29015d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal\" (UID: \"76002f047f028799fe97fbc79e29015d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:28.023930 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:28.023867    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:28.028195 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.028179    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76002f047f028799fe97fbc79e29015d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal\" (UID: \"76002f047f028799fe97fbc79e29015d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:28.028246 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.028219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76002f047f028799fe97fbc79e29015d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal\" (UID: \"76002f047f028799fe97fbc79e29015d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:28.028279 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.028242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae92e6bf8b06378903825aae1b8252ca-config\") pod \"kube-apiserver-proxy-ip-10-0-131-137.ec2.internal\" (UID: \"ae92e6bf8b06378903825aae1b8252ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:28.028329 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.028313    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76002f047f028799fe97fbc79e29015d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal\" (UID: \"76002f047f028799fe97fbc79e29015d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:28.028363 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.028326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76002f047f028799fe97fbc79e29015d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal\" (UID: \"76002f047f028799fe97fbc79e29015d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:28.028394 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.028369    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ae92e6bf8b06378903825aae1b8252ca-config\") pod \"kube-apiserver-proxy-ip-10-0-131-137.ec2.internal\" (UID: \"ae92e6bf8b06378903825aae1b8252ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:28.083352 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.083322    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:28.088378 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.088190    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:28.123987 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:28.123951    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:28.224540 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:28.224505    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:28.325081 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:28.325001    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:28.425462 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:28.425433    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:28.456816 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.456790    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:40:28.525734 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:28.525704    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:28.615408 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.615378    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:28.626187 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:28.626160    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:28.669434 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.669391    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:35:28 +0000 UTC" deadline="2028-02-19 12:06:02.68281692 +0000 UTC"
Sep 16 18:40:28.669434 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.669426    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12497h25m34.013394079s"
Sep 16 18:40:28.705511 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.705480    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:40:28.707010 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.706993    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:40:28.707098 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.707029    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:40:28.707098 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.707046    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:40:28.726913 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:28.726891    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:28.736755 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:28.736710    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae92e6bf8b06378903825aae1b8252ca.slice/crio-1d34fc13ccde0427cd625e342e0a6fe405e44a071db37f23fc998000f62f7a6c WatchSource:0}: Error finding container 1d34fc13ccde0427cd625e342e0a6fe405e44a071db37f23fc998000f62f7a6c: Status 404 returned error can't find the container with id 1d34fc13ccde0427cd625e342e0a6fe405e44a071db37f23fc998000f62f7a6c
Sep 16 18:40:28.737114 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:28.737094    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76002f047f028799fe97fbc79e29015d.slice/crio-6d5dff381bb2579aa8d8b0b9421608cfb0ecefc0eac20f13ab9080d03fd30774 WatchSource:0}: Error finding container 6d5dff381bb2579aa8d8b0b9421608cfb0ecefc0eac20f13ab9080d03fd30774: Status 404 returned error can't find the container with id 6d5dff381bb2579aa8d8b0b9421608cfb0ecefc0eac20f13ab9080d03fd30774
Sep 16 18:40:28.742190 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.742175    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:40:28.827771 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:28.827689    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:28.858987 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.858966    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:28.927992 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:28.927963    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:28.965395 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:28.965377    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:29.028307 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.028275    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-137.ec2.internal\" not found"
Sep 16 18:40:29.112449 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.112389    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:40:29.125308 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.125273    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:29.141181 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.141153    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:29.142142 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.142119    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-137.ec2.internal"
Sep 16 18:40:29.153566 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.153531    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:40:29.503557 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.503522    2583 apiserver.go:52] "Watching apiserver"
Sep 16 18:40:29.508892 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.508863    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:40:29.511021 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.510990    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2","openshift-cluster-node-tuning-operator/tuned-mv6mv","openshift-dns/node-resolver-gt9pt","openshift-multus/multus-additional-cni-plugins-nrtmh","openshift-multus/multus-nt66b","openshift-multus/network-metrics-daemon-n9vzt","openshift-network-diagnostics/network-check-target-qs4rw","openshift-network-operator/iptables-alerter-rkk6d","kube-system/global-pull-secret-syncer-7gt7h","kube-system/konnectivity-agent-m5v5w","kube-system/kube-apiserver-proxy-ip-10-0-131-137.ec2.internal","openshift-image-registry/node-ca-9tntl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-n9rcp"]
Sep 16 18:40:29.513753 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.513717    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:29.513877 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.513839    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:29.516166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.516145    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rkk6d"
Sep 16 18:40:29.519875 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.518642    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g89zc\""
Sep 16 18:40:29.519875 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.518925    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:29.519875 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.519116    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:29.519875 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.519279    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:40:29.519875 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.519565    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gt9pt"
Sep 16 18:40:29.520939 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.520921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.522231 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.522211    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-42vhc\""
Sep 16 18:40:29.522457 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.522441    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:40:29.522667 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.522650    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:40:29.522935 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.522915    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wmxd8\""
Sep 16 18:40:29.523126 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.523107    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:40:29.523486 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.523466    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:40:29.523721 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.523701    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:40:29.524270 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.523887    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:40:29.524270 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.524021    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:40:29.525713 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.525693    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.526147 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.526127    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:40:29.528862 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.528038    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:40:29.528862 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.528069    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-777c7\""
Sep 16 18:40:29.528862 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.528187    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:29.528862 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.528265    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:29.530628 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.530606    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:29.530728 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.530677    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:29.533861 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.533391    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.535528 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.535507    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:40:29.535743 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.535720    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:40:29.536067 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.536042    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:40:29.536258 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.536240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.536618 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.536598    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gbvkb\""
Sep 16 18:40:29.537194 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bef95704-b93c-4e72-b402-977ed172994e-host-slash\") pod \"iptables-alerter-rkk6d\" (UID: \"bef95704-b93c-4e72-b402-977ed172994e\") " pod="openshift-network-operator/iptables-alerter-rkk6d"
Sep 16 18:40:29.537290 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-hostroot\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.537290 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djrbv\" (UniqueName: \"kubernetes.io/projected/13bef125-8a64-4554-9ef9-fb4ddabc0628-kube-api-access-djrbv\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:29.537290 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38dd8c04-24c6-4a26-bdae-c9cb85d20453-system-cni-dir\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.537450 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-daemon-config\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.537450 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537311    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae2b5270-597d-4a80-8f88-908422bec8a9-hosts-file\") pod \"node-resolver-gt9pt\" (UID: \"ae2b5270-597d-4a80-8f88-908422bec8a9\") " pod="openshift-dns/node-resolver-gt9pt"
Sep 16 18:40:29.537450 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537336    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdfbw\" (UniqueName: \"kubernetes.io/projected/38dd8c04-24c6-4a26-bdae-c9cb85d20453-kube-api-access-gdfbw\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.537450 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-run-netns\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.537450 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537401    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:29.537450 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38dd8c04-24c6-4a26-bdae-c9cb85d20453-cnibin\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.537698 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537453    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38dd8c04-24c6-4a26-bdae-c9cb85d20453-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.537698 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38dd8c04-24c6-4a26-bdae-c9cb85d20453-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.537981 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.537537    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-socket-dir-parent\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.538071 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.538031    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-run-k8s-cni-cncf-io\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.538126 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.538082    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-var-lib-kubelet\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.538185 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.538162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2b5270-597d-4a80-8f88-908422bec8a9-tmp-dir\") pod \"node-resolver-gt9pt\" (UID: \"ae2b5270-597d-4a80-8f88-908422bec8a9\") " pod="openshift-dns/node-resolver-gt9pt"
Sep 16 18:40:29.538341 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.538315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h6cf\" (UniqueName: \"kubernetes.io/projected/bef95704-b93c-4e72-b402-977ed172994e-kube-api-access-9h6cf\") pod \"iptables-alerter-rkk6d\" (UID: \"bef95704-b93c-4e72-b402-977ed172994e\") " pod="openshift-network-operator/iptables-alerter-rkk6d"
Sep 16 18:40:29.538395 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.538370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2227\" (UniqueName: \"kubernetes.io/projected/ae2b5270-597d-4a80-8f88-908422bec8a9-kube-api-access-g2227\") pod \"node-resolver-gt9pt\" (UID: \"ae2b5270-597d-4a80-8f88-908422bec8a9\") " pod="openshift-dns/node-resolver-gt9pt"
Sep 16 18:40:29.538500 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.538406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38dd8c04-24c6-4a26-bdae-c9cb85d20453-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.538500 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.538440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-system-cni-dir\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.538500 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.538469    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87639fb2-3cf6-4d7c-b488-f00772b8f76b-cni-binary-copy\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.538638 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.538500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-conf-dir\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.538638 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.538530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcppw\" (UniqueName: \"kubernetes.io/projected/87639fb2-3cf6-4d7c-b488-f00772b8f76b-kube-api-access-zcppw\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.538738 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.538644    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:40:29.539255 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.539231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-var-lib-cni-bin\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.539369 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.539286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-var-lib-cni-multus\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.539369 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.539320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-run-multus-certs\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.539369 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.539339    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pw4tg\""
Sep 16 18:40:29.539369 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.539348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-etc-kubernetes\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.539571 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.539444    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:40:29.539622 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.539381    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xcr\" (UniqueName: \"kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr\") pod \"network-check-target-qs4rw\" (UID: \"a26f5576-408a-4cd3-892a-4261a45de399\") " pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:29.539679 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.539619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bef95704-b93c-4e72-b402-977ed172994e-iptables-alerter-script\") pod \"iptables-alerter-rkk6d\" (UID: \"bef95704-b93c-4e72-b402-977ed172994e\") " pod="openshift-network-operator/iptables-alerter-rkk6d"
Sep 16 18:40:29.540021 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.540002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38dd8c04-24c6-4a26-bdae-c9cb85d20453-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.540122 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.540064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-cni-dir\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.540122 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.540113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-cnibin\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.542082 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.542060    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:40:29.548218 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.548186    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:29.550855 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.550828    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tzrkn\""
Sep 16 18:40:29.550964 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.550862    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:40:29.550964 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.550886    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:40:29.551771 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.551579    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.553560 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.553535    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:40:29.553666 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.553586    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:40:29.553873 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.553853    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vg8fr\""
Sep 16 18:40:29.554442 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.554136    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9tntl"
Sep 16 18:40:29.554442 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.554181    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:40:29.554442 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.554197    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:40:29.554442 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.554209    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:40:29.554442 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.554243    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:40:29.558438 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.557944    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:40:29.558438 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.558118    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:40:29.558438 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.558136    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rbwd6\""
Sep 16 18:40:29.558438 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.558311    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:40:29.596894 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.596872    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l85jt"
Sep 16 18:40:29.604561 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.604538    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l85jt"
Sep 16 18:40:29.627584 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.627504    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:40:29.640336 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38dd8c04-24c6-4a26-bdae-c9cb85d20453-system-cni-dir\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.640474 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640343    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-var-lib-openvswitch\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.640474 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdfbw\" (UniqueName: \"kubernetes.io/projected/38dd8c04-24c6-4a26-bdae-c9cb85d20453-kube-api-access-gdfbw\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.640474 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-run-netns\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.640474 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-run-openvswitch\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.640474 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38dd8c04-24c6-4a26-bdae-c9cb85d20453-cnibin\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.640474 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-socket-dir-parent\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-run-k8s-cni-cncf-io\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-var-lib-kubelet\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-systemd-units\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640576    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-cni-netd\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640596    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-run-netns\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hfsr\" (UniqueName: \"kubernetes.io/projected/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-kube-api-access-9hfsr\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38dd8c04-24c6-4a26-bdae-c9cb85d20453-system-cni-dir\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2b5270-597d-4a80-8f88-908422bec8a9-tmp-dir\") pod \"node-resolver-gt9pt\" (UID: \"ae2b5270-597d-4a80-8f88-908422bec8a9\") " pod="openshift-dns/node-resolver-gt9pt"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640681    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-modprobe-d\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640701    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-run-ovn\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-log-socket\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3e6df42-134d-4588-bb47-5163ecb7925b-serviceca\") pod \"node-ca-9tntl\" (UID: \"c3e6df42-134d-4588-bb47-5163ecb7925b\") " pod="openshift-image-registry/node-ca-9tntl"
Sep 16 18:40:29.640800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640776    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbsdx\" (UniqueName: \"kubernetes.io/projected/c3e6df42-134d-4588-bb47-5163ecb7925b-kube-api-access-wbsdx\") pod \"node-ca-9tntl\" (UID: \"c3e6df42-134d-4588-bb47-5163ecb7925b\") " pod="openshift-image-registry/node-ca-9tntl"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-conf-dir\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640849    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-var-lib-kubelet\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-env-overrides\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-var-lib-cni-multus\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-etc-kubernetes\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640978    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b45a65e-e81b-45fe-8f11-db4a8685bc46-kubelet-config\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640990    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2b5270-597d-4a80-8f88-908422bec8a9-tmp-dir\") pod \"node-resolver-gt9pt\" (UID: \"ae2b5270-597d-4a80-8f88-908422bec8a9\") " pod="openshift-dns/node-resolver-gt9pt"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641102    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-conf-dir\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641168    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-var-lib-cni-multus\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641211    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-etc-kubernetes\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641309    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38dd8c04-24c6-4a26-bdae-c9cb85d20453-cnibin\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-socket-dir-parent\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641422    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-run-k8s-cni-cncf-io\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641458    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-var-lib-kubelet\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.640995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb2fq\" (UniqueName: \"kubernetes.io/projected/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-kube-api-access-qb2fq\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-kubernetes\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac692b25-53c6-46be-9cc4-fd7ba26529a2-tmp\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-etc-openvswitch\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3e6df42-134d-4588-bb47-5163ecb7925b-host\") pod \"node-ca-9tntl\" (UID: \"c3e6df42-134d-4588-bb47-5163ecb7925b\") " pod="openshift-image-registry/node-ca-9tntl"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xcr\" (UniqueName: \"kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr\") pod \"network-check-target-qs4rw\" (UID: \"a26f5576-408a-4cd3-892a-4261a45de399\") " pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bef95704-b93c-4e72-b402-977ed172994e-iptables-alerter-script\") pod \"iptables-alerter-rkk6d\" (UID: \"bef95704-b93c-4e72-b402-977ed172994e\") " pod="openshift-network-operator/iptables-alerter-rkk6d"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-cni-dir\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.641741 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-cni-dir\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-cnibin\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b45a65e-e81b-45fe-8f11-db4a8685bc46-dbus\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-socket-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-device-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641883    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-cnibin\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bef95704-b93c-4e72-b402-977ed172994e-host-slash\") pod \"iptables-alerter-rkk6d\" (UID: \"bef95704-b93c-4e72-b402-977ed172994e\") " pod="openshift-network-operator/iptables-alerter-rkk6d"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-etc-selinux\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-sysctl-conf\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bef95704-b93c-4e72-b402-977ed172994e-host-slash\") pod \"iptables-alerter-rkk6d\" (UID: \"bef95704-b93c-4e72-b402-977ed172994e\") " pod="openshift-network-operator/iptables-alerter-rkk6d"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-kubelet\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-run-systemd\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.641992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-ovnkube-config\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-daemon-config\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642174    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-tuned\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.642274 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bef95704-b93c-4e72-b402-977ed172994e-iptables-alerter-script\") pod \"iptables-alerter-rkk6d\" (UID: \"bef95704-b93c-4e72-b402-977ed172994e\") " pod="openshift-network-operator/iptables-alerter-rkk6d"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-run-netns\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-run-ovn-kubernetes\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae2b5270-597d-4a80-8f88-908422bec8a9-hosts-file\") pod \"node-resolver-gt9pt\" (UID: \"ae2b5270-597d-4a80-8f88-908422bec8a9\") " pod="openshift-dns/node-resolver-gt9pt"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642280    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a30106ce-a8b7-4549-b466-343652ad0736-agent-certs\") pod \"konnectivity-agent-m5v5w\" (UID: \"a30106ce-a8b7-4549-b466-343652ad0736\") " pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-cni-bin\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae2b5270-597d-4a80-8f88-908422bec8a9-hosts-file\") pod \"node-resolver-gt9pt\" (UID: \"ae2b5270-597d-4a80-8f88-908422bec8a9\") " pod="openshift-dns/node-resolver-gt9pt"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38dd8c04-24c6-4a26-bdae-c9cb85d20453-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642395    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38dd8c04-24c6-4a26-bdae-c9cb85d20453-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.642477    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.642544    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs podName:13bef125-8a64-4554-9ef9-fb4ddabc0628 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:30.142520713 +0000 UTC m=+3.025343801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs") pod "network-metrics-daemon-n9vzt" (UID: "13bef125-8a64-4554-9ef9-fb4ddabc0628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/87639fb2-3cf6-4d7c-b488-f00772b8f76b-multus-daemon-config\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-systemd\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-run\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a30106ce-a8b7-4549-b466-343652ad0736-konnectivity-ca\") pod \"konnectivity-agent-m5v5w\" (UID: \"a30106ce-a8b7-4549-b466-343652ad0736\") " pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-sysconfig\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.643111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-slash\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-node-log\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6cf\" (UniqueName: \"kubernetes.io/projected/bef95704-b93c-4e72-b402-977ed172994e-kube-api-access-9h6cf\") pod \"iptables-alerter-rkk6d\" (UID: \"bef95704-b93c-4e72-b402-977ed172994e\") " pod="openshift-network-operator/iptables-alerter-rkk6d"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642973    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2227\" (UniqueName: \"kubernetes.io/projected/ae2b5270-597d-4a80-8f88-908422bec8a9-kube-api-access-g2227\") pod \"node-resolver-gt9pt\" (UID: \"ae2b5270-597d-4a80-8f88-908422bec8a9\") " pod="openshift-dns/node-resolver-gt9pt"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.642981    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38dd8c04-24c6-4a26-bdae-c9cb85d20453-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643010    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38dd8c04-24c6-4a26-bdae-c9cb85d20453-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38dd8c04-24c6-4a26-bdae-c9cb85d20453-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-system-cni-dir\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87639fb2-3cf6-4d7c-b488-f00772b8f76b-cni-binary-copy\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcppw\" (UniqueName: \"kubernetes.io/projected/87639fb2-3cf6-4d7c-b488-f00772b8f76b-kube-api-access-zcppw\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-sys-fs\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-sys\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-var-lib-cni-bin\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-run-multus-certs\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-lib-modules\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-ovn-node-metrics-cert\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38dd8c04-24c6-4a26-bdae-c9cb85d20453-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.643678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643343    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-registration-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-sysctl-d\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-host\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-system-cni-dir\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-ovnkube-script-lib\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-hostroot\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643483    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crg7\" (UniqueName: \"kubernetes.io/projected/ac692b25-53c6-46be-9cc4-fd7ba26529a2-kube-api-access-6crg7\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djrbv\" (UniqueName: \"kubernetes.io/projected/13bef125-8a64-4554-9ef9-fb4ddabc0628-kube-api-access-djrbv\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38dd8c04-24c6-4a26-bdae-c9cb85d20453-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/87639fb2-3cf6-4d7c-b488-f00772b8f76b-cni-binary-copy\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643603    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-run-multus-certs\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643641    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-hostroot\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643663    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87639fb2-3cf6-4d7c-b488-f00772b8f76b-host-var-lib-cni-bin\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.644445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.643739    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38dd8c04-24c6-4a26-bdae-c9cb85d20453-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.650021 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.649989    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:29.650021 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.650015    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:29.650205 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.650031    2583 projected.go:196] Error preparing data for projected volume kube-api-access-79xcr for pod openshift-network-diagnostics/network-check-target-qs4rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:29.650205 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.650094    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr podName:a26f5576-408a-4cd3-892a-4261a45de399 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:30.150077073 +0000 UTC m=+3.032900153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-79xcr" (UniqueName: "kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr") pod "network-check-target-qs4rw" (UID: "a26f5576-408a-4cd3-892a-4261a45de399") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:29.650355 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.650181    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:40:29.655530 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.655504    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2227\" (UniqueName: \"kubernetes.io/projected/ae2b5270-597d-4a80-8f88-908422bec8a9-kube-api-access-g2227\") pod \"node-resolver-gt9pt\" (UID: \"ae2b5270-597d-4a80-8f88-908422bec8a9\") " pod="openshift-dns/node-resolver-gt9pt"
Sep 16 18:40:29.655951 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.655930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcppw\" (UniqueName: \"kubernetes.io/projected/87639fb2-3cf6-4d7c-b488-f00772b8f76b-kube-api-access-zcppw\") pod \"multus-nt66b\" (UID: \"87639fb2-3cf6-4d7c-b488-f00772b8f76b\") " pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.656065 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.655993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6cf\" (UniqueName: \"kubernetes.io/projected/bef95704-b93c-4e72-b402-977ed172994e-kube-api-access-9h6cf\") pod \"iptables-alerter-rkk6d\" (UID: \"bef95704-b93c-4e72-b402-977ed172994e\") " pod="openshift-network-operator/iptables-alerter-rkk6d"
Sep 16 18:40:29.656897 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.656860    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdfbw\" (UniqueName: \"kubernetes.io/projected/38dd8c04-24c6-4a26-bdae-c9cb85d20453-kube-api-access-gdfbw\") pod \"multus-additional-cni-plugins-nrtmh\" (UID: \"38dd8c04-24c6-4a26-bdae-c9cb85d20453\") " pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.658259 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.658214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djrbv\" (UniqueName: \"kubernetes.io/projected/13bef125-8a64-4554-9ef9-fb4ddabc0628-kube-api-access-djrbv\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:29.658259 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.658214    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal" event={"ID":"76002f047f028799fe97fbc79e29015d","Type":"ContainerStarted","Data":"6d5dff381bb2579aa8d8b0b9421608cfb0ecefc0eac20f13ab9080d03fd30774"}
Sep 16 18:40:29.661569 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.661542    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-137.ec2.internal" event={"ID":"ae92e6bf8b06378903825aae1b8252ca","Type":"ContainerStarted","Data":"1d34fc13ccde0427cd625e342e0a6fe405e44a071db37f23fc998000f62f7a6c"}
Sep 16 18:40:29.743896 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.743818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:29.743896 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.743852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.743896 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.743869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-tuned\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.743896 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.743886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-run-netns\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.744213 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.743980    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.744213 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.744002    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:29.744213 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-run-ovn-kubernetes\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.744213 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a30106ce-a8b7-4549-b466-343652ad0736-agent-certs\") pod \"konnectivity-agent-m5v5w\" (UID: \"a30106ce-a8b7-4549-b466-343652ad0736\") " pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:29.744213 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:29.744087    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret podName:2b45a65e-e81b-45fe-8f11-db4a8685bc46 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:30.244065292 +0000 UTC m=+3.126888376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret") pod "global-pull-secret-syncer-7gt7h" (UID: "2b45a65e-e81b-45fe-8f11-db4a8685bc46") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:29.744213 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744203    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-run-netns\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-run-ovn-kubernetes\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744287    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-cni-bin\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-systemd\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-run\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a30106ce-a8b7-4549-b466-343652ad0736-konnectivity-ca\") pod \"konnectivity-agent-m5v5w\" (UID: \"a30106ce-a8b7-4549-b466-343652ad0736\") " pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-sysconfig\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-slash\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-node-log\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744410    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-systemd\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-sys-fs\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744460    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-node-log\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-sys\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.744522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-lib-modules\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-ovn-node-metrics-cert\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-registration-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744574    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-run\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-sysctl-d\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-sysconfig\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-host\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744661    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-cni-bin\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744692    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-ovnkube-script-lib\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744698    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-sys-fs\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crg7\" (UniqueName: \"kubernetes.io/projected/ac692b25-53c6-46be-9cc4-fd7ba26529a2-kube-api-access-6crg7\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-var-lib-openvswitch\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-run-openvswitch\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-systemd-units\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-cni-netd\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfsr\" (UniqueName: \"kubernetes.io/projected/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-kube-api-access-9hfsr\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-modprobe-d\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.745166 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745084    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-run-ovn\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-log-socket\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3e6df42-134d-4588-bb47-5163ecb7925b-serviceca\") pod \"node-ca-9tntl\" (UID: \"c3e6df42-134d-4588-bb47-5163ecb7925b\") " pod="openshift-image-registry/node-ca-9tntl"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbsdx\" (UniqueName: \"kubernetes.io/projected/c3e6df42-134d-4588-bb47-5163ecb7925b-kube-api-access-wbsdx\") pod \"node-ca-9tntl\" (UID: \"c3e6df42-134d-4588-bb47-5163ecb7925b\") " pod="openshift-image-registry/node-ca-9tntl"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-var-lib-kubelet\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-env-overrides\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b45a65e-e81b-45fe-8f11-db4a8685bc46-kubelet-config\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb2fq\" (UniqueName: \"kubernetes.io/projected/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-kube-api-access-qb2fq\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745345    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-kubernetes\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-ovnkube-script-lib\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac692b25-53c6-46be-9cc4-fd7ba26529a2-tmp\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-etc-openvswitch\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3e6df42-134d-4588-bb47-5163ecb7925b-host\") pod \"node-ca-9tntl\" (UID: \"c3e6df42-134d-4588-bb47-5163ecb7925b\") " pod="openshift-image-registry/node-ca-9tntl"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b45a65e-e81b-45fe-8f11-db4a8685bc46-dbus\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-socket-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-device-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.745690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-etc-selinux\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-sysctl-conf\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-kubelet\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-run-systemd\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-ovnkube-config\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-registration-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.745962    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-sysctl-d\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746013    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-host\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.744617    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-slash\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746035    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-ovnkube-config\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746067    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-run-ovn\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746083    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-etc-openvswitch\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746113    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-sys\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746126    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c3e6df42-134d-4588-bb47-5163ecb7925b-host\") pod \"node-ca-9tntl\" (UID: \"c3e6df42-134d-4588-bb47-5163ecb7925b\") " pod="openshift-image-registry/node-ca-9tntl"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b45a65e-e81b-45fe-8f11-db4a8685bc46-dbus\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746268    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-socket-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-device-dir\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-etc-selinux\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.746761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746524    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-sysctl-conf\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746572    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-kubelet\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-run-systemd\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a30106ce-a8b7-4549-b466-343652ad0736-konnectivity-ca\") pod \"konnectivity-agent-m5v5w\" (UID: \"a30106ce-a8b7-4549-b466-343652ad0736\") " pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-log-socket\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-lib-modules\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-var-lib-openvswitch\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746788    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-run-openvswitch\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746849    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-systemd-units\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.746885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-host-cni-netd\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.747009    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-tuned\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.747077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a30106ce-a8b7-4549-b466-343652ad0736-agent-certs\") pod \"konnectivity-agent-m5v5w\" (UID: \"a30106ce-a8b7-4549-b466-343652ad0736\") " pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.747143    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-modprobe-d\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.747171    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-var-lib-kubelet\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.747192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c3e6df42-134d-4588-bb47-5163ecb7925b-serviceca\") pod \"node-ca-9tntl\" (UID: \"c3e6df42-134d-4588-bb47-5163ecb7925b\") " pod="openshift-image-registry/node-ca-9tntl"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.747199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b45a65e-e81b-45fe-8f11-db4a8685bc46-kubelet-config\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.747253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-ovn-node-metrics-cert\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.747703 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.747269    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac692b25-53c6-46be-9cc4-fd7ba26529a2-etc-kubernetes\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.748646 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.747598    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-env-overrides\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.748751 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.748726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac692b25-53c6-46be-9cc4-fd7ba26529a2-tmp\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.757021 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.756996    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crg7\" (UniqueName: \"kubernetes.io/projected/ac692b25-53c6-46be-9cc4-fd7ba26529a2-kube-api-access-6crg7\") pod \"tuned-mv6mv\" (UID: \"ac692b25-53c6-46be-9cc4-fd7ba26529a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.759458 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.759429    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbsdx\" (UniqueName: \"kubernetes.io/projected/c3e6df42-134d-4588-bb47-5163ecb7925b-kube-api-access-wbsdx\") pod \"node-ca-9tntl\" (UID: \"c3e6df42-134d-4588-bb47-5163ecb7925b\") " pod="openshift-image-registry/node-ca-9tntl"
Sep 16 18:40:29.759619 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.759602    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb2fq\" (UniqueName: \"kubernetes.io/projected/fb9dadcf-b4c0-4959-b872-b097fdfeea3a-kube-api-access-qb2fq\") pod \"aws-ebs-csi-driver-node-ktsr2\" (UID: \"fb9dadcf-b4c0-4959-b872-b097fdfeea3a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.760005 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.759989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfsr\" (UniqueName: \"kubernetes.io/projected/351b8c67-9d1b-4e1f-9e3d-f108f1f39058-kube-api-access-9hfsr\") pod \"ovnkube-node-n9rcp\" (UID: \"351b8c67-9d1b-4e1f-9e3d-f108f1f39058\") " pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.831157 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.831132    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rkk6d"
Sep 16 18:40:29.851162 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.851135    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gt9pt"
Sep 16 18:40:29.857881 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.857858    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nrtmh"
Sep 16 18:40:29.865796 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.865770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nt66b"
Sep 16 18:40:29.873801 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.873773    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2"
Sep 16 18:40:29.880621 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.880604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mv6mv"
Sep 16 18:40:29.889546 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.889525    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:29.897522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.897498    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:29.902351 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:29.902330    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9tntl"
Sep 16 18:40:30.148146 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:30.148068    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:30.148302 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:30.148255    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:30.148362 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:30.148343    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs podName:13bef125-8a64-4554-9ef9-fb4ddabc0628 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:31.148319493 +0000 UTC m=+4.031142570 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs") pod "network-metrics-daemon-n9vzt" (UID: "13bef125-8a64-4554-9ef9-fb4ddabc0628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:30.249190 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:30.249155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xcr\" (UniqueName: \"kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr\") pod \"network-check-target-qs4rw\" (UID: \"a26f5576-408a-4cd3-892a-4261a45de399\") " pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:30.249380 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:30.249204    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:30.249380 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:30.249342    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:30.249380 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:30.249355    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:30.249380 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:30.249368    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:30.249380 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:30.249381    2583 projected.go:196] Error preparing data for projected volume kube-api-access-79xcr for pod openshift-network-diagnostics/network-check-target-qs4rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:30.249625 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:30.249420    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret podName:2b45a65e-e81b-45fe-8f11-db4a8685bc46 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:31.249398916 +0000 UTC m=+4.132221993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret") pod "global-pull-secret-syncer-7gt7h" (UID: "2b45a65e-e81b-45fe-8f11-db4a8685bc46") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:30.249625 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:30.249440    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr podName:a26f5576-408a-4cd3-892a-4261a45de399 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:31.249429704 +0000 UTC m=+4.132252765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-79xcr" (UniqueName: "kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr") pod "network-check-target-qs4rw" (UID: "a26f5576-408a-4cd3-892a-4261a45de399") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:30.609083 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:30.609047    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:30 +0000 UTC" deadline="2028-02-11 04:53:30.351987288 +0000 UTC"
Sep 16 18:40:30.609083 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:30.609079    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12298h12m59.742911848s"
Sep 16 18:40:30.652926 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:30.652892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:30.653061 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:30.653002    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:30.686267 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:30.686197    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351b8c67_9d1b_4e1f_9e3d_f108f1f39058.slice/crio-8d39b96bf3aeb7fa8d07865cfd1c592f4ced78f736af0791248cf6bcb59c46bb WatchSource:0}: Error finding container 8d39b96bf3aeb7fa8d07865cfd1c592f4ced78f736af0791248cf6bcb59c46bb: Status 404 returned error can't find the container with id 8d39b96bf3aeb7fa8d07865cfd1c592f4ced78f736af0791248cf6bcb59c46bb
Sep 16 18:40:30.689256 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:30.689177    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e6df42_134d_4588_bb47_5163ecb7925b.slice/crio-9af359108c2fdf070e11a810d3708be4c0697d8fc779f60d0ddc81a1edc77cbe WatchSource:0}: Error finding container 9af359108c2fdf070e11a810d3708be4c0697d8fc779f60d0ddc81a1edc77cbe: Status 404 returned error can't find the container with id 9af359108c2fdf070e11a810d3708be4c0697d8fc779f60d0ddc81a1edc77cbe
Sep 16 18:40:30.692108 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:30.692088    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2b5270_597d_4a80_8f88_908422bec8a9.slice/crio-dc2083306271e8e01a6cf17c0845c45e33c5a91ac2c5bb28ed31ec4ee000d8cd WatchSource:0}: Error finding container dc2083306271e8e01a6cf17c0845c45e33c5a91ac2c5bb28ed31ec4ee000d8cd: Status 404 returned error can't find the container with id dc2083306271e8e01a6cf17c0845c45e33c5a91ac2c5bb28ed31ec4ee000d8cd
Sep 16 18:40:30.696147 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:30.696125    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef95704_b93c_4e72_b402_977ed172994e.slice/crio-90ebec5e6b0922c616b14ff0fa159a472807a945dc9394e9b6a377923588c8f5 WatchSource:0}: Error finding container 90ebec5e6b0922c616b14ff0fa159a472807a945dc9394e9b6a377923588c8f5: Status 404 returned error can't find the container with id 90ebec5e6b0922c616b14ff0fa159a472807a945dc9394e9b6a377923588c8f5
Sep 16 18:40:30.697839 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:30.697729    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87639fb2_3cf6_4d7c_b488_f00772b8f76b.slice/crio-9e6546cb9268b3533ae58b8534eceaab5e91601233167c3a2ec993e85a021573 WatchSource:0}: Error finding container 9e6546cb9268b3533ae58b8534eceaab5e91601233167c3a2ec993e85a021573: Status 404 returned error can't find the container with id 9e6546cb9268b3533ae58b8534eceaab5e91601233167c3a2ec993e85a021573
Sep 16 18:40:30.698632 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:30.698611    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac692b25_53c6_46be_9cc4_fd7ba26529a2.slice/crio-b278d63a8fd284c3f73252fda32bfa76993ee15f9f0431fcf9fb6c4c5e14a492 WatchSource:0}: Error finding container b278d63a8fd284c3f73252fda32bfa76993ee15f9f0431fcf9fb6c4c5e14a492: Status 404 returned error can't find the container with id b278d63a8fd284c3f73252fda32bfa76993ee15f9f0431fcf9fb6c4c5e14a492
Sep 16 18:40:30.700620 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:30.699888    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30106ce_a8b7_4549_b466_343652ad0736.slice/crio-ca624bc7a87e25bee704f5401b56a4387fee380ec6e7072bb022cd2c231a43b0 WatchSource:0}: Error finding container ca624bc7a87e25bee704f5401b56a4387fee380ec6e7072bb022cd2c231a43b0: Status 404 returned error can't find the container with id ca624bc7a87e25bee704f5401b56a4387fee380ec6e7072bb022cd2c231a43b0
Sep 16 18:40:30.702156 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:30.701838    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9dadcf_b4c0_4959_b872_b097fdfeea3a.slice/crio-a57a52893d027b83543f53f351dc639be3dff185bf3e16f841db689e5242ee4f WatchSource:0}: Error finding container a57a52893d027b83543f53f351dc639be3dff185bf3e16f841db689e5242ee4f: Status 404 returned error can't find the container with id a57a52893d027b83543f53f351dc639be3dff185bf3e16f841db689e5242ee4f
Sep 16 18:40:30.702847 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:40:30.702718    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dd8c04_24c6_4a26_bdae_c9cb85d20453.slice/crio-839540a95b5e57aa8b159c5443989b4882d8dae3f58cbdddf816dd17857b471e WatchSource:0}: Error finding container 839540a95b5e57aa8b159c5443989b4882d8dae3f58cbdddf816dd17857b471e: Status 404 returned error can't find the container with id 839540a95b5e57aa8b159c5443989b4882d8dae3f58cbdddf816dd17857b471e
Sep 16 18:40:31.157740 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.157153    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:31.157740 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:31.157322    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:31.157740 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:31.157388    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs podName:13bef125-8a64-4554-9ef9-fb4ddabc0628 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.157368333 +0000 UTC m=+6.040191406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs") pod "network-metrics-daemon-n9vzt" (UID: "13bef125-8a64-4554-9ef9-fb4ddabc0628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:31.258135 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.258094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:31.258389 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.258202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xcr\" (UniqueName: \"kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr\") pod \"network-check-target-qs4rw\" (UID: \"a26f5576-408a-4cd3-892a-4261a45de399\") " pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:31.258389 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:31.258346    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:31.258389 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:31.258368    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:31.258389 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:31.258382    2583 projected.go:196] Error preparing data for projected volume kube-api-access-79xcr for pod openshift-network-diagnostics/network-check-target-qs4rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:31.258611 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:31.258446    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr podName:a26f5576-408a-4cd3-892a-4261a45de399 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.258427082 +0000 UTC m=+6.141250146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-79xcr" (UniqueName: "kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr") pod "network-check-target-qs4rw" (UID: "a26f5576-408a-4cd3-892a-4261a45de399") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:31.259215 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:31.258899    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:31.259215 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:31.258962    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret podName:2b45a65e-e81b-45fe-8f11-db4a8685bc46 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.258945416 +0000 UTC m=+6.141768483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret") pod "global-pull-secret-syncer-7gt7h" (UID: "2b45a65e-e81b-45fe-8f11-db4a8685bc46") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:31.610103 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.609987    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:35:30 +0000 UTC" deadline="2028-02-14 22:06:00.801884437 +0000 UTC"
Sep 16 18:40:31.610103 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.610053    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12387h25m29.191836962s"
Sep 16 18:40:31.655293 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.655261    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:31.655451 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:31.655410    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:31.655878 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.655855    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:31.655993 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:31.655970    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:31.679202 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.679128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rkk6d" event={"ID":"bef95704-b93c-4e72-b402-977ed172994e","Type":"ContainerStarted","Data":"90ebec5e6b0922c616b14ff0fa159a472807a945dc9394e9b6a377923588c8f5"}
Sep 16 18:40:31.684076 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.684018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9tntl" event={"ID":"c3e6df42-134d-4588-bb47-5163ecb7925b","Type":"ContainerStarted","Data":"9af359108c2fdf070e11a810d3708be4c0697d8fc779f60d0ddc81a1edc77cbe"}
Sep 16 18:40:31.697244 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.697216    2583 generic.go:356] "Generic (PLEG): container finished" podID="76002f047f028799fe97fbc79e29015d" containerID="42f7283c2a0215085df4966dc2ab5f4febcf2f6384cad69442cd90ae9b85d864" exitCode=0
Sep 16 18:40:31.697363 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.697301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal" event={"ID":"76002f047f028799fe97fbc79e29015d","Type":"ContainerDied","Data":"42f7283c2a0215085df4966dc2ab5f4febcf2f6384cad69442cd90ae9b85d864"}
Sep 16 18:40:31.738125 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.738091    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-137.ec2.internal" event={"ID":"ae92e6bf8b06378903825aae1b8252ca","Type":"ContainerStarted","Data":"558ef757327cd3ba004998e21a0ffa522588bb3c72cba5c03b7dd0d7ae07bd2f"}
Sep 16 18:40:31.754222 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.754047    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrtmh" event={"ID":"38dd8c04-24c6-4a26-bdae-c9cb85d20453","Type":"ContainerStarted","Data":"839540a95b5e57aa8b159c5443989b4882d8dae3f58cbdddf816dd17857b471e"}
Sep 16 18:40:31.763186 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.763077    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2" event={"ID":"fb9dadcf-b4c0-4959-b872-b097fdfeea3a","Type":"ContainerStarted","Data":"a57a52893d027b83543f53f351dc639be3dff185bf3e16f841db689e5242ee4f"}
Sep 16 18:40:31.782112 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.782046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5v5w" event={"ID":"a30106ce-a8b7-4549-b466-343652ad0736","Type":"ContainerStarted","Data":"ca624bc7a87e25bee704f5401b56a4387fee380ec6e7072bb022cd2c231a43b0"}
Sep 16 18:40:31.795240 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.795204    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gt9pt" event={"ID":"ae2b5270-597d-4a80-8f88-908422bec8a9","Type":"ContainerStarted","Data":"dc2083306271e8e01a6cf17c0845c45e33c5a91ac2c5bb28ed31ec4ee000d8cd"}
Sep 16 18:40:31.805859 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.805803    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp" event={"ID":"351b8c67-9d1b-4e1f-9e3d-f108f1f39058","Type":"ContainerStarted","Data":"8d39b96bf3aeb7fa8d07865cfd1c592f4ced78f736af0791248cf6bcb59c46bb"}
Sep 16 18:40:31.808243 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.808034    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mv6mv" event={"ID":"ac692b25-53c6-46be-9cc4-fd7ba26529a2","Type":"ContainerStarted","Data":"b278d63a8fd284c3f73252fda32bfa76993ee15f9f0431fcf9fb6c4c5e14a492"}
Sep 16 18:40:31.815855 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:31.815824    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nt66b" event={"ID":"87639fb2-3cf6-4d7c-b488-f00772b8f76b","Type":"ContainerStarted","Data":"9e6546cb9268b3533ae58b8534eceaab5e91601233167c3a2ec993e85a021573"}
Sep 16 18:40:32.653402 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:32.653365    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:32.653940 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:32.653506    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:32.857718 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:32.857676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal" event={"ID":"76002f047f028799fe97fbc79e29015d","Type":"ContainerStarted","Data":"16134988a727d54935f4f61d80a4206e5011130d9881530bef76a45006285819"}
Sep 16 18:40:32.875052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:32.874994    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-137.ec2.internal" podStartSLOduration=3.8749739290000003 podStartE2EDuration="3.874973929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:31.756396735 +0000 UTC m=+4.639219842" watchObservedRunningTime="2026-09-16 18:40:32.874973929 +0000 UTC m=+5.757797018"
Sep 16 18:40:32.875249 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:32.875138    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-137.ec2.internal" podStartSLOduration=3.875131976 podStartE2EDuration="3.875131976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:32.874511192 +0000 UTC m=+5.757334280" watchObservedRunningTime="2026-09-16 18:40:32.875131976 +0000 UTC m=+5.757955062"
Sep 16 18:40:33.174412 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:33.173802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:33.174412 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:33.173984    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:33.174412 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:33.174058    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs podName:13bef125-8a64-4554-9ef9-fb4ddabc0628 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:37.17403756 +0000 UTC m=+10.056860624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs") pod "network-metrics-daemon-n9vzt" (UID: "13bef125-8a64-4554-9ef9-fb4ddabc0628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:33.274438 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:33.274402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xcr\" (UniqueName: \"kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr\") pod \"network-check-target-qs4rw\" (UID: \"a26f5576-408a-4cd3-892a-4261a45de399\") " pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:33.274616 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:33.274463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:33.274696 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:33.274630    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:33.274749 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:33.274695    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret podName:2b45a65e-e81b-45fe-8f11-db4a8685bc46 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:37.274671956 +0000 UTC m=+10.157495018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret") pod "global-pull-secret-syncer-7gt7h" (UID: "2b45a65e-e81b-45fe-8f11-db4a8685bc46") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:33.275164 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:33.275144    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:33.275265 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:33.275173    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:33.275265 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:33.275187    2583 projected.go:196] Error preparing data for projected volume kube-api-access-79xcr for pod openshift-network-diagnostics/network-check-target-qs4rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:33.275265 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:33.275235    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr podName:a26f5576-408a-4cd3-892a-4261a45de399 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:37.275219992 +0000 UTC m=+10.158043059 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-79xcr" (UniqueName: "kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr") pod "network-check-target-qs4rw" (UID: "a26f5576-408a-4cd3-892a-4261a45de399") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:33.656178 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:33.656085    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:33.656672 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:33.656231    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:33.656846 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:33.656702    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:33.656846 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:33.656802    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:34.653544 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:34.653506    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:34.653733 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:34.653671    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:35.653409 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:35.652893    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:35.653409 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:35.653045    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:35.653409 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:35.653106    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:35.653409 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:35.653269    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:36.652933 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:36.652573    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:36.653099 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:36.653007    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:37.211174 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:37.211131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:37.211655 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:37.211304    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:37.211655 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:37.211390    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs podName:13bef125-8a64-4554-9ef9-fb4ddabc0628 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:45.211356844 +0000 UTC m=+18.094179922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs") pod "network-metrics-daemon-n9vzt" (UID: "13bef125-8a64-4554-9ef9-fb4ddabc0628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:37.312065 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:37.312021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xcr\" (UniqueName: \"kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr\") pod \"network-check-target-qs4rw\" (UID: \"a26f5576-408a-4cd3-892a-4261a45de399\") " pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:37.312255 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:37.312097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:37.312324 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:37.312254    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:37.312324 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:37.312305    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret podName:2b45a65e-e81b-45fe-8f11-db4a8685bc46 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:45.312291315 +0000 UTC m=+18.195114390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret") pod "global-pull-secret-syncer-7gt7h" (UID: "2b45a65e-e81b-45fe-8f11-db4a8685bc46") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:37.312644 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:37.312620    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:37.312720 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:37.312652    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:37.312720 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:37.312665    2583 projected.go:196] Error preparing data for projected volume kube-api-access-79xcr for pod openshift-network-diagnostics/network-check-target-qs4rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:37.312720 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:37.312717    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr podName:a26f5576-408a-4cd3-892a-4261a45de399 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:45.312699928 +0000 UTC m=+18.195523010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-79xcr" (UniqueName: "kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr") pod "network-check-target-qs4rw" (UID: "a26f5576-408a-4cd3-892a-4261a45de399") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:37.653804 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:37.653728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:37.654027 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:37.653871    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:37.654329 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:37.654313    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:37.654398 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:37.654384    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:38.653577 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:38.653142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:38.653577 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:38.653255    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:39.655246 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:39.654647    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:39.655246 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:39.654768    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:39.655246 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:39.655121    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:39.655246 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:39.655199    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:40.653460 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:40.653430    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:40.653623 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:40.653543    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:41.652783 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:41.652753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:41.653197 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:41.652859    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:41.653197 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:41.652948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:41.653197 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:41.653063    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:42.655150 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:42.655114    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:42.655620 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:42.655245    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:43.653334 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:43.653296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:43.653334 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:43.653318    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:43.653560 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:43.653438    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:43.653614 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:43.653591    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:44.653032 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:44.653001    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:44.653459 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:44.653124    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:45.269632 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:45.269603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:45.269821 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:45.269728    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:45.269821 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:45.269802    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs podName:13bef125-8a64-4554-9ef9-fb4ddabc0628 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:01.269787724 +0000 UTC m=+34.152610783 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs") pod "network-metrics-daemon-n9vzt" (UID: "13bef125-8a64-4554-9ef9-fb4ddabc0628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:45.370806 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:45.370780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:45.370918 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:45.370841    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xcr\" (UniqueName: \"kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr\") pod \"network-check-target-qs4rw\" (UID: \"a26f5576-408a-4cd3-892a-4261a45de399\") " pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:45.370964 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:45.370928    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:45.370964 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:45.370942    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:45.370964 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:45.370958    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:45.371051 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:45.370969    2583 projected.go:196] Error preparing data for projected volume kube-api-access-79xcr for pod openshift-network-diagnostics/network-check-target-qs4rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:45.371051 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:45.370989    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret podName:2b45a65e-e81b-45fe-8f11-db4a8685bc46 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:01.370972424 +0000 UTC m=+34.253795486 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret") pod "global-pull-secret-syncer-7gt7h" (UID: "2b45a65e-e81b-45fe-8f11-db4a8685bc46") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:45.371051 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:45.371006    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr podName:a26f5576-408a-4cd3-892a-4261a45de399 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:01.370995527 +0000 UTC m=+34.253818604 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-79xcr" (UniqueName: "kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr") pod "network-check-target-qs4rw" (UID: "a26f5576-408a-4cd3-892a-4261a45de399") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:45.652833 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:45.652732    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:45.652978 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:45.652882    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:45.652978 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:45.652948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:45.653107 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:45.653065    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:46.653556 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:46.653523    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:46.653950 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:46.653625    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:47.654234 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:47.654195    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:47.654662 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:47.654308    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:47.654662 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:47.654368    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:47.654662 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:47.654429    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:48.653391 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:48.653365    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:48.653539 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:48.653467    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:49.653249 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.653027    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:49.653859 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.653104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:49.653859 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:49.653350    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:49.653859 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:49.653408    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:49.889955 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.889924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nt66b" event={"ID":"87639fb2-3cf6-4d7c-b488-f00772b8f76b","Type":"ContainerStarted","Data":"cb575ce3c32e475948f37726b89d702a0388bcee2d115593219d477e7418f7bb"}
Sep 16 18:40:49.891656 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.891624    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9tntl" event={"ID":"c3e6df42-134d-4588-bb47-5163ecb7925b","Type":"ContainerStarted","Data":"1b3079554d6d71328d711834f9b38aa14ab38644e318f2f3a13f4203da29fdff"}
Sep 16 18:40:49.893278 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.893257    2583 generic.go:356] "Generic (PLEG): container finished" podID="38dd8c04-24c6-4a26-bdae-c9cb85d20453" containerID="325432ff7b4aaa0ae59c5af4082141ab6451bcdce196a4d829737d192c11b494" exitCode=0
Sep 16 18:40:49.893383 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.893326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrtmh" event={"ID":"38dd8c04-24c6-4a26-bdae-c9cb85d20453","Type":"ContainerDied","Data":"325432ff7b4aaa0ae59c5af4082141ab6451bcdce196a4d829737d192c11b494"}
Sep 16 18:40:49.895536 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.895244    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2" event={"ID":"fb9dadcf-b4c0-4959-b872-b097fdfeea3a","Type":"ContainerStarted","Data":"9cee18116ee8162d806b40e3326b617cb5730a272ac726daea8331fe3414a16f"}
Sep 16 18:40:49.896869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.896845    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5v5w" event={"ID":"a30106ce-a8b7-4549-b466-343652ad0736","Type":"ContainerStarted","Data":"ce560f0630245a1155924b99bd12e5fe2e15ecbe9dbc768b4a3143d2f1eaa33c"}
Sep 16 18:40:49.898695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.898676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gt9pt" event={"ID":"ae2b5270-597d-4a80-8f88-908422bec8a9","Type":"ContainerStarted","Data":"800cf107661bb8edaafd3a3ae224f41015afab28c5d4a900577568f75ce901b3"}
Sep 16 18:40:49.902208 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.902185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp" event={"ID":"351b8c67-9d1b-4e1f-9e3d-f108f1f39058","Type":"ContainerStarted","Data":"cd42ad2753b2cfb7051c4d61e5330c7eb0f16a2b12fa07a4e444c35be1b17fc6"}
Sep 16 18:40:49.902290 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.902258    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp" event={"ID":"351b8c67-9d1b-4e1f-9e3d-f108f1f39058","Type":"ContainerStarted","Data":"d6074a0da0cc1dd30cfb2a830eece571a1678f97358bcd528e574ab6081b4e29"}
Sep 16 18:40:49.902333 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.902312    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp" event={"ID":"351b8c67-9d1b-4e1f-9e3d-f108f1f39058","Type":"ContainerStarted","Data":"e33fe68b5ef0ae856da470018272c5a0c79139ee136765c3a039457d0ad30b74"}
Sep 16 18:40:49.902377 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.902360    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp" event={"ID":"351b8c67-9d1b-4e1f-9e3d-f108f1f39058","Type":"ContainerStarted","Data":"40ff7612a10f2cd6e746a29780b27f2841b35f02efc386a2612bdf848bdc8c4f"}
Sep 16 18:40:49.902483 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.902409    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp" event={"ID":"351b8c67-9d1b-4e1f-9e3d-f108f1f39058","Type":"ContainerStarted","Data":"ac4347c9d0dcaca2a1351b16663232b4cb05875b49ab78c8e62339a72127aaa8"}
Sep 16 18:40:49.902483 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.902451    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp" event={"ID":"351b8c67-9d1b-4e1f-9e3d-f108f1f39058","Type":"ContainerStarted","Data":"d5c1d559a0cc29d9aa2d16710a3b47e179e208792337cf9bc24a15f442a59805"}
Sep 16 18:40:49.903976 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.903922    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mv6mv" event={"ID":"ac692b25-53c6-46be-9cc4-fd7ba26529a2","Type":"ContainerStarted","Data":"580b33bb6dbeec79e3517414fda8b66f9e4d0de98b3018cac7a83b37d3182a94"}
Sep 16 18:40:49.913316 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.913274    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nt66b" podStartSLOduration=3.400287021 podStartE2EDuration="21.913258171s" totalImagesPullingTime="18.51297115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:49.910312171 +0000 UTC m=+22.793135253" watchObservedRunningTime="2026-09-16 18:40:49.913258171 +0000 UTC m=+22.796081252"
Sep 16 18:40:49.943957 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.943915    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gt9pt" podStartSLOduration=3.481625817 podStartE2EDuration="21.94389892s" totalImagesPullingTime="18.462273103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:49.943833487 +0000 UTC m=+22.826656573" watchObservedRunningTime="2026-09-16 18:40:49.94389892 +0000 UTC m=+22.826722003"
Sep 16 18:40:49.944333 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.944308    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9tntl" podStartSLOduration=3.482135871 podStartE2EDuration="21.944278526s" totalImagesPullingTime="18.462142655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:49.928067236 +0000 UTC m=+22.810890317" watchObservedRunningTime="2026-09-16 18:40:49.944278526 +0000 UTC m=+22.827101788"
Sep 16 18:40:49.988074 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.988036    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mv6mv" podStartSLOduration=3.532057629 podStartE2EDuration="21.988023822s" totalImagesPullingTime="18.455966193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:49.987970331 +0000 UTC m=+22.870793411" watchObservedRunningTime="2026-09-16 18:40:49.988023822 +0000 UTC m=+22.870846904"
Sep 16 18:40:49.991806 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:49.991792    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:50.004840 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:50.004595    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m5v5w" podStartSLOduration=10.827284678 podStartE2EDuration="22.004578023s" totalImagesPullingTime="11.177293345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:50.003821677 +0000 UTC m=+22.886644759" watchObservedRunningTime="2026-09-16 18:40:50.004578023 +0000 UTC m=+22.887401108"
Sep 16 18:40:50.564456 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:50.564427    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:40:50.596516 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:50.596433    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:40:50.564453164Z","UUID":"6fc68025-f5e1-4209-82f4-b8c8885f3cb2","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:40:50.598146 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:50.598119    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:40:50.598255 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:50.598153    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:40:50.652593 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:50.652570    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:50.652711 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:50.652666    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:50.909332 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:50.909247    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2" event={"ID":"fb9dadcf-b4c0-4959-b872-b097fdfeea3a","Type":"ContainerStarted","Data":"4931159337d4456f50e7c909716599568ebd60860dc7c56c46179f5f37c91376"}
Sep 16 18:40:50.911185 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:50.911150    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rkk6d" event={"ID":"bef95704-b93c-4e72-b402-977ed172994e","Type":"ContainerStarted","Data":"816fecbc624549ec01934a3de152e873c888b22177a4e4b5444b696d39aecc2a"}
Sep 16 18:40:50.930419 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:50.930363    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rkk6d" podStartSLOduration=4.471985238 podStartE2EDuration="22.930346587s" totalImagesPullingTime="18.458361349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:50.930236595 +0000 UTC m=+23.813059676" watchObservedRunningTime="2026-09-16 18:40:50.930346587 +0000 UTC m=+23.813169669"
Sep 16 18:40:51.653056 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:51.653020    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:51.653214 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:51.653099    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:51.653278 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:51.653223    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:51.653586 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:51.653562    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:51.917787 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:51.916920    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2" event={"ID":"fb9dadcf-b4c0-4959-b872-b097fdfeea3a","Type":"ContainerStarted","Data":"20e606a029749cc60f99f13f1079d5fe8a89b162ecd3e90ff6bee4718b195c96"}
Sep 16 18:40:51.921155 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:51.921128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp" event={"ID":"351b8c67-9d1b-4e1f-9e3d-f108f1f39058","Type":"ContainerStarted","Data":"9a6536fe6adfcfe88ddfb0b8bc26ce45837b0d034d30771aabdb6f8d67187725"}
Sep 16 18:40:51.944323 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:51.944280    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ktsr2" podStartSLOduration=3.495749914 podStartE2EDuration="23.944265474s" totalImagesPullingTime="20.44851556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:51.943818211 +0000 UTC m=+24.826641314" watchObservedRunningTime="2026-09-16 18:40:51.944265474 +0000 UTC m=+24.827088568"
Sep 16 18:40:52.652863 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:52.652827    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:52.653056 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:52.652935    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:52.661996 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:52.661965    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:52.662575 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:52.662551    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:52.923990 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:52.923908    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m5v5w"
Sep 16 18:40:53.652822 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:53.652774    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:53.652822 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:53.652818    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:53.653094 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:53.652903    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:53.653094 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:53.653057    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:54.652979 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:54.652835    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:54.653455 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:54.653059    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:54.930383 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:54.930353    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp" event={"ID":"351b8c67-9d1b-4e1f-9e3d-f108f1f39058","Type":"ContainerStarted","Data":"b8817f398a8d3963ed1a2549c8e31784bb41e982fbbab34e31d26a49b9df47f0"}
Sep 16 18:40:54.930690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:54.930664    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:54.932309 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:54.932284    2583 generic.go:356] "Generic (PLEG): container finished" podID="38dd8c04-24c6-4a26-bdae-c9cb85d20453" containerID="97e68f9402e3e3135dbe6e9ffdf37e5efb07518e4e42b9d54a3649d2c73a289b" exitCode=0
Sep 16 18:40:54.932410 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:54.932355    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrtmh" event={"ID":"38dd8c04-24c6-4a26-bdae-c9cb85d20453","Type":"ContainerDied","Data":"97e68f9402e3e3135dbe6e9ffdf37e5efb07518e4e42b9d54a3649d2c73a289b"}
Sep 16 18:40:54.946528 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:54.946504    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:54.964391 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:54.964350    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp" podStartSLOduration=8.43994112 podStartE2EDuration="26.964339154s" totalImagesPullingTime="18.524398034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:54.963904239 +0000 UTC m=+27.846727320" watchObservedRunningTime="2026-09-16 18:40:54.964339154 +0000 UTC m=+27.847162259"
Sep 16 18:40:55.653561 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:55.653523    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:55.654064 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:55.653636    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:55.654064 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:55.653726    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:55.654064 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:55.653848    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:55.935949 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:55.935906    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:55.936184 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:55.936007    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:55.953037 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:55.953008    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:40:56.258017 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:56.257953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7gt7h"]
Sep 16 18:40:56.258140 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:56.258066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:56.258178 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:56.258154    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:56.260702 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:56.260680    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qs4rw"]
Sep 16 18:40:56.260831 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:56.260770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:56.260875 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:56.260834    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:56.263366 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:56.263344    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n9vzt"]
Sep 16 18:40:56.263446 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:56.263434    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:56.263561 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:56.263540    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:56.940159 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:56.939925    2583 generic.go:356] "Generic (PLEG): container finished" podID="38dd8c04-24c6-4a26-bdae-c9cb85d20453" containerID="65bca120f8a647b6b6108b8cac9fee88be3b26376a997c7393a9576e50bccb5d" exitCode=0
Sep 16 18:40:56.940490 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:56.940056    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrtmh" event={"ID":"38dd8c04-24c6-4a26-bdae-c9cb85d20453","Type":"ContainerDied","Data":"65bca120f8a647b6b6108b8cac9fee88be3b26376a997c7393a9576e50bccb5d"}
Sep 16 18:40:57.653869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:57.653842    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:57.654052 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:57.653942    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:40:57.654052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:57.654035    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:57.654171 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:57.654129    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:58.652845 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:58.652816    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:40:58.653194 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:58.652918    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:40:58.947281 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:58.947249    2583 generic.go:356] "Generic (PLEG): container finished" podID="38dd8c04-24c6-4a26-bdae-c9cb85d20453" containerID="6e2ab252304cd53565fc9fe097b3e04374773ea4501344873969ebf78a582aca" exitCode=0
Sep 16 18:40:58.947396 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:58.947323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrtmh" event={"ID":"38dd8c04-24c6-4a26-bdae-c9cb85d20453","Type":"ContainerDied","Data":"6e2ab252304cd53565fc9fe097b3e04374773ea4501344873969ebf78a582aca"}
Sep 16 18:40:59.652980 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:59.652947    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:40:59.653426 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:59.653080    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:40:59.653426 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:40:59.653149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:40:59.653426 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:40:59.653245    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:41:00.653468 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:00.653429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:41:00.653891 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:00.653551    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vzt" podUID="13bef125-8a64-4554-9ef9-fb4ddabc0628"
Sep 16 18:41:01.289152 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.289113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:41:01.289401 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.289252    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:01.289401 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.289325    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs podName:13bef125-8a64-4554-9ef9-fb4ddabc0628 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.289304355 +0000 UTC m=+66.172127418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs") pod "network-metrics-daemon-n9vzt" (UID: "13bef125-8a64-4554-9ef9-fb4ddabc0628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:41:01.389653 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.389621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xcr\" (UniqueName: \"kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr\") pod \"network-check-target-qs4rw\" (UID: \"a26f5576-408a-4cd3-892a-4261a45de399\") " pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:41:01.389846 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.389659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:41:01.389846 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.389775    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:01.389846 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.389812    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:41:01.389846 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.389833    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret podName:2b45a65e-e81b-45fe-8f11-db4a8685bc46 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.389817581 +0000 UTC m=+66.272640660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret") pod "global-pull-secret-syncer-7gt7h" (UID: "2b45a65e-e81b-45fe-8f11-db4a8685bc46") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:41:01.389846 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.389838    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:41:01.389846 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.389852    2583 projected.go:196] Error preparing data for projected volume kube-api-access-79xcr for pod openshift-network-diagnostics/network-check-target-qs4rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:01.390079 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.389909    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr podName:a26f5576-408a-4cd3-892a-4261a45de399 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.389892055 +0000 UTC m=+66.272715115 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-79xcr" (UniqueName: "kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr") pod "network-check-target-qs4rw" (UID: "a26f5576-408a-4cd3-892a-4261a45de399") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:41:01.653211 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.653133    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:41:01.653355 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.653270    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gt7h" podUID="2b45a65e-e81b-45fe-8f11-db4a8685bc46"
Sep 16 18:41:01.653355 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.653330    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:41:01.653467 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.653441    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs4rw" podUID="a26f5576-408a-4cd3-892a-4261a45de399"
Sep 16 18:41:01.743902 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.743872    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-137.ec2.internal" event="NodeReady"
Sep 16 18:41:01.744285 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.744005    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:41:01.779761 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.779711    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fb8d7c95b-djcg5"]
Sep 16 18:41:01.783898 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.783874    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.786247 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.786227    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:41:01.786381 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.786363    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:41:01.786453 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.786228    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:41:01.786662 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.786646    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2ndk7\""
Sep 16 18:41:01.790727 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.790703    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:41:01.795570 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.795531    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb8d7c95b-djcg5"]
Sep 16 18:41:01.797903 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.797882    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2qbqd"]
Sep 16 18:41:01.800908 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.800887    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5nt49"]
Sep 16 18:41:01.801034 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.801007    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:01.802969 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.802952    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:41:01.803048 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.802987    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zm88x\""
Sep 16 18:41:01.803048 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.802997    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:41:01.803521 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.803505    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5nt49"
Sep 16 18:41:01.805438 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.805418    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:41:01.805530 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.805440    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:41:01.805587 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.805539    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:41:01.805760 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.805724    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2kfsb\""
Sep 16 18:41:01.810828 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.810807    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2qbqd"]
Sep 16 18:41:01.813925 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.813906    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5nt49"]
Sep 16 18:41:01.894395 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87eb20c3-96fb-47c9-9b08-b4b4042a374b-ca-trust-extracted\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.894395 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894397    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-trusted-ca\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.894594 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb3b2b9-dea0-4346-9418-9b9394b13d82-tmp-dir\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:01.894594 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hr27\" (UniqueName: \"kubernetes.io/projected/aeb3b2b9-dea0-4346-9418-9b9394b13d82-kube-api-access-2hr27\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:01.894594 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.894594 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-bound-sa-token\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.894594 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7x75\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-kube-api-access-m7x75\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.894820 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894620    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-image-registry-private-configuration\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.894820 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-installation-pull-secrets\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.894820 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894659    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9947aa6-6269-453c-9a91-e95f47fffac3-cert\") pod \"ingress-canary-5nt49\" (UID: \"f9947aa6-6269-453c-9a91-e95f47fffac3\") " pod="openshift-ingress-canary/ingress-canary-5nt49"
Sep 16 18:41:01.894820 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:01.894820 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnpdc\" (UniqueName: \"kubernetes.io/projected/f9947aa6-6269-453c-9a91-e95f47fffac3-kube-api-access-pnpdc\") pod \"ingress-canary-5nt49\" (UID: \"f9947aa6-6269-453c-9a91-e95f47fffac3\") " pod="openshift-ingress-canary/ingress-canary-5nt49"
Sep 16 18:41:01.894820 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894736    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-certificates\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.894820 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.894773    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb3b2b9-dea0-4346-9418-9b9394b13d82-config-volume\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:01.995657 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.995630    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-trusted-ca\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.995657 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.995659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb3b2b9-dea0-4346-9418-9b9394b13d82-tmp-dir\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:01.995898 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.995681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hr27\" (UniqueName: \"kubernetes.io/projected/aeb3b2b9-dea0-4346-9418-9b9394b13d82-kube-api-access-2hr27\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:01.995898 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.995713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.995898 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.995788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-bound-sa-token\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.996173 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.996150    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:01.996173 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.996175    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb8d7c95b-djcg5: secret "image-registry-tls" not found
Sep 16 18:41:01.996302 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.996253    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls podName:87eb20c3-96fb-47c9-9b08-b4b4042a374b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:02.496231041 +0000 UTC m=+35.379054104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls") pod "image-registry-fb8d7c95b-djcg5" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b") : secret "image-registry-tls" not found
Sep 16 18:41:01.996368 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.996346    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeb3b2b9-dea0-4346-9418-9b9394b13d82-tmp-dir\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:01.996695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.996655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7x75\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-kube-api-access-m7x75\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.998695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.997529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-image-registry-private-configuration\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.998695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.997539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-trusted-ca\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.998695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.997734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-installation-pull-secrets\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.998695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.997828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9947aa6-6269-453c-9a91-e95f47fffac3-cert\") pod \"ingress-canary-5nt49\" (UID: \"f9947aa6-6269-453c-9a91-e95f47fffac3\") " pod="openshift-ingress-canary/ingress-canary-5nt49"
Sep 16 18:41:01.998695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.997982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:01.998695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.998026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnpdc\" (UniqueName: \"kubernetes.io/projected/f9947aa6-6269-453c-9a91-e95f47fffac3-kube-api-access-pnpdc\") pod \"ingress-canary-5nt49\" (UID: \"f9947aa6-6269-453c-9a91-e95f47fffac3\") " pod="openshift-ingress-canary/ingress-canary-5nt49"
Sep 16 18:41:01.998695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.998075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-certificates\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.998695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.998225    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb3b2b9-dea0-4346-9418-9b9394b13d82-config-volume\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:01.998695 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.998239    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:01.998695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.998265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87eb20c3-96fb-47c9-9b08-b4b4042a374b-ca-trust-extracted\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:01.998695 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:01.998372    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls podName:aeb3b2b9-dea0-4346-9418-9b9394b13d82 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:02.498282475 +0000 UTC m=+35.381105550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls") pod "dns-default-2qbqd" (UID: "aeb3b2b9-dea0-4346-9418-9b9394b13d82") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:01.999269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.998953    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87eb20c3-96fb-47c9-9b08-b4b4042a374b-ca-trust-extracted\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:02.000821 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.999693    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-certificates\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:02.000821 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:01.999823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb3b2b9-dea0-4346-9418-9b9394b13d82-config-volume\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:02.002372 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.002349    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-image-registry-private-configuration\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:02.003652 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.003628    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-installation-pull-secrets\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:02.003802 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.003782    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9947aa6-6269-453c-9a91-e95f47fffac3-cert\") pod \"ingress-canary-5nt49\" (UID: \"f9947aa6-6269-453c-9a91-e95f47fffac3\") " pod="openshift-ingress-canary/ingress-canary-5nt49"
Sep 16 18:41:02.010027 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.009833    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7x75\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-kube-api-access-m7x75\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:02.010027 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.009868    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnpdc\" (UniqueName: \"kubernetes.io/projected/f9947aa6-6269-453c-9a91-e95f47fffac3-kube-api-access-pnpdc\") pod \"ingress-canary-5nt49\" (UID: \"f9947aa6-6269-453c-9a91-e95f47fffac3\") " pod="openshift-ingress-canary/ingress-canary-5nt49"
Sep 16 18:41:02.010027 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.009989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-bound-sa-token\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:02.010328 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.010307    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hr27\" (UniqueName: \"kubernetes.io/projected/aeb3b2b9-dea0-4346-9418-9b9394b13d82-kube-api-access-2hr27\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:02.118857 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.118829    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5nt49"
Sep 16 18:41:02.503169 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.503134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:02.503359 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.503229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:02.503359 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:02.503317    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:02.503359 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:02.503341    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb8d7c95b-djcg5: secret "image-registry-tls" not found
Sep 16 18:41:02.503359 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:02.503345    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:02.503537 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:02.503408    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls podName:aeb3b2b9-dea0-4346-9418-9b9394b13d82 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:03.50339 +0000 UTC m=+36.386213074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls") pod "dns-default-2qbqd" (UID: "aeb3b2b9-dea0-4346-9418-9b9394b13d82") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:02.503537 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:02.503427    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls podName:87eb20c3-96fb-47c9-9b08-b4b4042a374b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:03.503417429 +0000 UTC m=+36.386240492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls") pod "image-registry-fb8d7c95b-djcg5" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b") : secret "image-registry-tls" not found
Sep 16 18:41:02.653573 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.653547    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:41:02.655972 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.655942    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:02.656221 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:02.656201    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx4q5\""
Sep 16 18:41:03.153276 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.153076    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5nt49"]
Sep 16 18:41:03.163509 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:41:03.163483    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9947aa6_6269_453c_9a91_e95f47fffac3.slice/crio-75f2df1cc5a8e54cb1014cb64c44611709aeb29b0de3e1bba65a9aff2e93e0a0 WatchSource:0}: Error finding container 75f2df1cc5a8e54cb1014cb64c44611709aeb29b0de3e1bba65a9aff2e93e0a0: Status 404 returned error can't find the container with id 75f2df1cc5a8e54cb1014cb64c44611709aeb29b0de3e1bba65a9aff2e93e0a0
Sep 16 18:41:03.510805 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.510769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:03.510942 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.510831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:03.510942 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:03.510922    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:03.511009 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:03.510976    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls podName:aeb3b2b9-dea0-4346-9418-9b9394b13d82 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:05.51096392 +0000 UTC m=+38.393786983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls") pod "dns-default-2qbqd" (UID: "aeb3b2b9-dea0-4346-9418-9b9394b13d82") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:03.511009 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:03.510925    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:03.511078 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:03.511016    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb8d7c95b-djcg5: secret "image-registry-tls" not found
Sep 16 18:41:03.511078 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:03.511056    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls podName:87eb20c3-96fb-47c9-9b08-b4b4042a374b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:05.511043998 +0000 UTC m=+38.393867073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls") pod "image-registry-fb8d7c95b-djcg5" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b") : secret "image-registry-tls" not found
Sep 16 18:41:03.653244 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.653212    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:41:03.653436 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.653418    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:41:03.656548 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.656526    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:03.657154 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.657134    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:03.658262 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.658240    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:03.658367 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.658327    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lk5bf\""
Sep 16 18:41:03.962154 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.962117    2583 generic.go:356] "Generic (PLEG): container finished" podID="38dd8c04-24c6-4a26-bdae-c9cb85d20453" containerID="dad319206d28eb81836ab3be9964b6cea0fbb7b6dac1a2fe4337defc45864f7e" exitCode=0
Sep 16 18:41:03.962345 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.962201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrtmh" event={"ID":"38dd8c04-24c6-4a26-bdae-c9cb85d20453","Type":"ContainerDied","Data":"dad319206d28eb81836ab3be9964b6cea0fbb7b6dac1a2fe4337defc45864f7e"}
Sep 16 18:41:03.964931 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:03.964879    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5nt49" event={"ID":"f9947aa6-6269-453c-9a91-e95f47fffac3","Type":"ContainerStarted","Data":"75f2df1cc5a8e54cb1014cb64c44611709aeb29b0de3e1bba65a9aff2e93e0a0"}
Sep 16 18:41:04.971023 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:04.970987    2583 generic.go:356] "Generic (PLEG): container finished" podID="38dd8c04-24c6-4a26-bdae-c9cb85d20453" containerID="ba7252f389c87864cee2de995d078683f55152659be08d5db812ded511761a4e" exitCode=0
Sep 16 18:41:04.971727 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:04.971062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrtmh" event={"ID":"38dd8c04-24c6-4a26-bdae-c9cb85d20453","Type":"ContainerDied","Data":"ba7252f389c87864cee2de995d078683f55152659be08d5db812ded511761a4e"}
Sep 16 18:41:05.526546 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:05.526313    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:05.526546 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:05.526465    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:05.526546 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:05.526510    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:05.526546 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:05.526530    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls podName:aeb3b2b9-dea0-4346-9418-9b9394b13d82 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:09.526515112 +0000 UTC m=+42.409338176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls") pod "dns-default-2qbqd" (UID: "aeb3b2b9-dea0-4346-9418-9b9394b13d82") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:05.526803 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:05.526587    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:05.526803 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:05.526597    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb8d7c95b-djcg5: secret "image-registry-tls" not found
Sep 16 18:41:05.526803 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:05.526649    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls podName:87eb20c3-96fb-47c9-9b08-b4b4042a374b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:09.526635231 +0000 UTC m=+42.409458297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls") pod "image-registry-fb8d7c95b-djcg5" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b") : secret "image-registry-tls" not found
Sep 16 18:41:05.977220 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:05.977188    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrtmh" event={"ID":"38dd8c04-24c6-4a26-bdae-c9cb85d20453","Type":"ContainerStarted","Data":"8e6ef460b0dfafaec7030a2b081ec1ba1eb753dde8355618ff56d15b51b493aa"}
Sep 16 18:41:05.978688 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:05.978663    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5nt49" event={"ID":"f9947aa6-6269-453c-9a91-e95f47fffac3","Type":"ContainerStarted","Data":"54f60ba3e41f06d288f3d0b99daa709285da5dab513e219498ea9d0b705f38ef"}
Sep 16 18:41:06.002564 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:06.002517    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nrtmh" podStartSLOduration=8.369980195 podStartE2EDuration="38.00250161s" totalImagesPullingTime="29.632521415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:06.001172193 +0000 UTC m=+38.883995276" watchObservedRunningTime="2026-09-16 18:41:06.00250161 +0000 UTC m=+38.885324692"
Sep 16 18:41:06.019697 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:06.019665    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5nt49" podStartSLOduration=2.919171633 podStartE2EDuration="5.019652792s" totalImagesPullingTime="2.100481159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:06.018759798 +0000 UTC m=+38.901582878" watchObservedRunningTime="2026-09-16 18:41:06.019652792 +0000 UTC m=+38.902475873"
Sep 16 18:41:09.550995 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:09.550957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:09.551364 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:09.551010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:09.551364 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:09.551103    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:09.551364 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:09.551113    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb8d7c95b-djcg5: secret "image-registry-tls" not found
Sep 16 18:41:09.551364 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:09.551117    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:09.551364 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:09.551160    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls podName:87eb20c3-96fb-47c9-9b08-b4b4042a374b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:17.551147879 +0000 UTC m=+50.433970938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls") pod "image-registry-fb8d7c95b-djcg5" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b") : secret "image-registry-tls" not found
Sep 16 18:41:09.551364 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:09.551172    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls podName:aeb3b2b9-dea0-4346-9418-9b9394b13d82 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:17.5511662 +0000 UTC m=+50.433989258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls") pod "dns-default-2qbqd" (UID: "aeb3b2b9-dea0-4346-9418-9b9394b13d82") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:17.608727 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:17.608695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:17.609145 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:17.608784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:17.609145 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:17.608843    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:17.609145 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:17.608864    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb8d7c95b-djcg5: secret "image-registry-tls" not found
Sep 16 18:41:17.609145 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:17.608864    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:17.609145 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:17.608911    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls podName:aeb3b2b9-dea0-4346-9418-9b9394b13d82 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.608899061 +0000 UTC m=+66.491722119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls") pod "dns-default-2qbqd" (UID: "aeb3b2b9-dea0-4346-9418-9b9394b13d82") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:17.609145 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:17.608923    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls podName:87eb20c3-96fb-47c9-9b08-b4b4042a374b nodeName:}" failed. No retries permitted until 2026-09-16 18:41:33.608916968 +0000 UTC m=+66.491740026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls") pod "image-registry-fb8d7c95b-djcg5" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b") : secret "image-registry-tls" not found
Sep 16 18:41:27.953551 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:27.953525    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n9rcp"
Sep 16 18:41:33.314792 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.314759    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:41:33.321494 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.321475    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:41:33.325754 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:33.325720    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:41:33.325847 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:33.325826    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs podName:13bef125-8a64-4554-9ef9-fb4ddabc0628 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:37.325801516 +0000 UTC m=+130.208624575 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs") pod "network-metrics-daemon-n9vzt" (UID: "13bef125-8a64-4554-9ef9-fb4ddabc0628") : secret "metrics-daemon-secret" not found
Sep 16 18:41:33.415076 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.415051    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xcr\" (UniqueName: \"kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr\") pod \"network-check-target-qs4rw\" (UID: \"a26f5576-408a-4cd3-892a-4261a45de399\") " pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:41:33.415204 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.415094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:41:33.417780 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.417754    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:41:33.417780 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.417762    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:41:33.427971 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.427946    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:41:33.429043 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.429025    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b45a65e-e81b-45fe-8f11-db4a8685bc46-original-pull-secret\") pod \"global-pull-secret-syncer-7gt7h\" (UID: \"2b45a65e-e81b-45fe-8f11-db4a8685bc46\") " pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:41:33.439004 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.438983    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xcr\" (UniqueName: \"kubernetes.io/projected/a26f5576-408a-4cd3-892a-4261a45de399-kube-api-access-79xcr\") pod \"network-check-target-qs4rw\" (UID: \"a26f5576-408a-4cd3-892a-4261a45de399\") " pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:41:33.616378 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.616319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:41:33.616378 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.616359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:41:33.616509 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:33.616448    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:33.616509 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:33.616458    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb8d7c95b-djcg5: secret "image-registry-tls" not found
Sep 16 18:41:33.616509 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:33.616457    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:33.616601 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:33.616515    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls podName:87eb20c3-96fb-47c9-9b08-b4b4042a374b nodeName:}" failed. No retries permitted until 2026-09-16 18:42:05.616503116 +0000 UTC m=+98.499326174 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls") pod "image-registry-fb8d7c95b-djcg5" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b") : secret "image-registry-tls" not found
Sep 16 18:41:33.616601 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:41:33.616526    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls podName:aeb3b2b9-dea0-4346-9418-9b9394b13d82 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:05.616520921 +0000 UTC m=+98.499343980 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls") pod "dns-default-2qbqd" (UID: "aeb3b2b9-dea0-4346-9418-9b9394b13d82") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:33.668884 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.668862    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lk5bf\""
Sep 16 18:41:33.672555 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.672540    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gt7h"
Sep 16 18:41:33.677081 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.677066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:41:33.988498 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:33.988456    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7gt7h"]
Sep 16 18:41:33.991907 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:41:33.991884    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b45a65e_e81b_45fe_8f11_db4a8685bc46.slice/crio-3d23b253fd0313a1a4a9a095d9c4e2efbe8e08e75f16932b58f817103275befb WatchSource:0}: Error finding container 3d23b253fd0313a1a4a9a095d9c4e2efbe8e08e75f16932b58f817103275befb: Status 404 returned error can't find the container with id 3d23b253fd0313a1a4a9a095d9c4e2efbe8e08e75f16932b58f817103275befb
Sep 16 18:41:34.016596 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:34.016573    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qs4rw"]
Sep 16 18:41:34.019706 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:41:34.019682    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26f5576_408a_4cd3_892a_4261a45de399.slice/crio-ccedb969e5b2becaba0330d574216b2bcf6595698dc227dbe9876fc2b864b0b7 WatchSource:0}: Error finding container ccedb969e5b2becaba0330d574216b2bcf6595698dc227dbe9876fc2b864b0b7: Status 404 returned error can't find the container with id ccedb969e5b2becaba0330d574216b2bcf6595698dc227dbe9876fc2b864b0b7
Sep 16 18:41:34.043442 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:34.043418    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qs4rw" event={"ID":"a26f5576-408a-4cd3-892a-4261a45de399","Type":"ContainerStarted","Data":"ccedb969e5b2becaba0330d574216b2bcf6595698dc227dbe9876fc2b864b0b7"}
Sep 16 18:41:34.044561 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:34.044538    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7gt7h" event={"ID":"2b45a65e-e81b-45fe-8f11-db4a8685bc46","Type":"ContainerStarted","Data":"3d23b253fd0313a1a4a9a095d9c4e2efbe8e08e75f16932b58f817103275befb"}
Sep 16 18:41:39.059521 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:39.059482    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qs4rw" event={"ID":"a26f5576-408a-4cd3-892a-4261a45de399","Type":"ContainerStarted","Data":"ca129f6e9865f5c8db4269a2ce759e6964c3e7311d3279efa3a9b4c7e8ae25cd"}
Sep 16 18:41:39.060003 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:39.059571    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:41:39.061071 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:39.061040    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7gt7h" event={"ID":"2b45a65e-e81b-45fe-8f11-db4a8685bc46","Type":"ContainerStarted","Data":"578575ca0b6fc1bbc038a3fb752964d60fd5e8a72433d237cc7190fe6e027d95"}
Sep 16 18:41:39.084710 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:39.084669    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qs4rw" podStartSLOduration=66.711990404 podStartE2EDuration="1m11.08465758s" totalImagesPullingTime="4.372667176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:39.083980043 +0000 UTC m=+71.966803124" watchObservedRunningTime="2026-09-16 18:41:39.08465758 +0000 UTC m=+71.967480657"
Sep 16 18:41:39.099410 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:41:39.099369    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7gt7h" podStartSLOduration=66.694543593 podStartE2EDuration="1m11.099353772s" totalImagesPullingTime="4.404810179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:39.099109518 +0000 UTC m=+71.981932597" watchObservedRunningTime="2026-09-16 18:41:39.099353772 +0000 UTC m=+71.982176853"
Sep 16 18:42:05.623322 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:05.623214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls\") pod \"image-registry-fb8d7c95b-djcg5\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") " pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:42:05.623322 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:05.623274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:42:05.623946 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:05.623375    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:42:05.623946 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:05.623379    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:42:05.623946 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:05.623395    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb8d7c95b-djcg5: secret "image-registry-tls" not found
Sep 16 18:42:05.623946 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:05.623462    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls podName:87eb20c3-96fb-47c9-9b08-b4b4042a374b nodeName:}" failed. No retries permitted until 2026-09-16 18:43:09.623444839 +0000 UTC m=+162.506267899 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls") pod "image-registry-fb8d7c95b-djcg5" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b") : secret "image-registry-tls" not found
Sep 16 18:42:05.623946 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:05.623478    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls podName:aeb3b2b9-dea0-4346-9418-9b9394b13d82 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:09.623471532 +0000 UTC m=+162.506294591 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls") pod "dns-default-2qbqd" (UID: "aeb3b2b9-dea0-4346-9418-9b9394b13d82") : secret "dns-default-metrics-tls" not found
Sep 16 18:42:10.066835 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.066802    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qs4rw"
Sep 16 18:42:10.618141 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.618109    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-jcr5s"]
Sep 16 18:42:10.656603 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.656576    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-jcr5s"]
Sep 16 18:42:10.656758 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.656655    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.661027 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.660997    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:42:10.661027 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.661021    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 18:42:10.661218 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.661106    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:42:10.661218 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.661197    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5lfv5\""
Sep 16 18:42:10.661334 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.661244    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 18:42:10.669881 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.669862    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 18:42:10.730666 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.730646    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-fww7m"]
Sep 16 18:42:10.744639 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.744618    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fww7m"
Sep 16 18:42:10.746932 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.746910    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zpszq\""
Sep 16 18:42:10.752797 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.752777    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-fww7m"]
Sep 16 18:42:10.758345 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.758325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7761e9ff-e826-4316-9a1a-876df7e23311-service-ca-bundle\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.758433 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.758352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7761e9ff-e826-4316-9a1a-876df7e23311-serving-cert\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.758433 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.758378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7761e9ff-e826-4316-9a1a-876df7e23311-snapshots\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.758503 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.758445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7761e9ff-e826-4316-9a1a-876df7e23311-trusted-ca-bundle\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.758565 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.758545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7761e9ff-e826-4316-9a1a-876df7e23311-tmp\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.758607 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.758583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6ltl\" (UniqueName: \"kubernetes.io/projected/7761e9ff-e826-4316-9a1a-876df7e23311-kube-api-access-l6ltl\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.859687 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.859667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6ltl\" (UniqueName: \"kubernetes.io/projected/7761e9ff-e826-4316-9a1a-876df7e23311-kube-api-access-l6ltl\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.859834 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.859708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7761e9ff-e826-4316-9a1a-876df7e23311-service-ca-bundle\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.859834 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.859726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7761e9ff-e826-4316-9a1a-876df7e23311-serving-cert\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.859834 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.859774    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57jwx\" (UniqueName: \"kubernetes.io/projected/bbd239e1-4420-4b27-bf19-2e29fcfec3f2-kube-api-access-57jwx\") pod \"network-check-source-6f8b768455-fww7m\" (UID: \"bbd239e1-4420-4b27-bf19-2e29fcfec3f2\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-fww7m"
Sep 16 18:42:10.859937 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.859906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7761e9ff-e826-4316-9a1a-876df7e23311-snapshots\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.859970 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.859937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7761e9ff-e826-4316-9a1a-876df7e23311-trusted-ca-bundle\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.860018 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.859985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7761e9ff-e826-4316-9a1a-876df7e23311-tmp\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.860365 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.860333    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7761e9ff-e826-4316-9a1a-876df7e23311-service-ca-bundle\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.860365 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.860357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7761e9ff-e826-4316-9a1a-876df7e23311-tmp\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.860704 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.860586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7761e9ff-e826-4316-9a1a-876df7e23311-snapshots\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.860789 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.860704    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7761e9ff-e826-4316-9a1a-876df7e23311-trusted-ca-bundle\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.862299 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.862278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7761e9ff-e826-4316-9a1a-876df7e23311-serving-cert\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.870090 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.870043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6ltl\" (UniqueName: \"kubernetes.io/projected/7761e9ff-e826-4316-9a1a-876df7e23311-kube-api-access-l6ltl\") pod \"insights-operator-5844b856db-jcr5s\" (UID: \"7761e9ff-e826-4316-9a1a-876df7e23311\") " pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.961149 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.961112    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jwx\" (UniqueName: \"kubernetes.io/projected/bbd239e1-4420-4b27-bf19-2e29fcfec3f2-kube-api-access-57jwx\") pod \"network-check-source-6f8b768455-fww7m\" (UID: \"bbd239e1-4420-4b27-bf19-2e29fcfec3f2\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-fww7m"
Sep 16 18:42:10.965181 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.965164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-jcr5s"
Sep 16 18:42:10.971004 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:10.970975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57jwx\" (UniqueName: \"kubernetes.io/projected/bbd239e1-4420-4b27-bf19-2e29fcfec3f2-kube-api-access-57jwx\") pod \"network-check-source-6f8b768455-fww7m\" (UID: \"bbd239e1-4420-4b27-bf19-2e29fcfec3f2\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-fww7m"
Sep 16 18:42:11.052585 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:11.052543    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fww7m"
Sep 16 18:42:11.244623 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:11.244592    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-jcr5s"]
Sep 16 18:42:11.248257 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:11.248232    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7761e9ff_e826_4316_9a1a_876df7e23311.slice/crio-a80424b4a74d5dc6e708011152e6acdb0875ac5c7788cdaaeb1df7162d8b62a0 WatchSource:0}: Error finding container a80424b4a74d5dc6e708011152e6acdb0875ac5c7788cdaaeb1df7162d8b62a0: Status 404 returned error can't find the container with id a80424b4a74d5dc6e708011152e6acdb0875ac5c7788cdaaeb1df7162d8b62a0
Sep 16 18:42:11.332539 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:11.332513    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-fww7m"]
Sep 16 18:42:11.334934 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:11.334910    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd239e1_4420_4b27_bf19_2e29fcfec3f2.slice/crio-b45bdc38dcea25ce71aae346f65155b3e3ef761189815191368a9ef2fc66ad9a WatchSource:0}: Error finding container b45bdc38dcea25ce71aae346f65155b3e3ef761189815191368a9ef2fc66ad9a: Status 404 returned error can't find the container with id b45bdc38dcea25ce71aae346f65155b3e3ef761189815191368a9ef2fc66ad9a
Sep 16 18:42:12.142325 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:12.142289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fww7m" event={"ID":"bbd239e1-4420-4b27-bf19-2e29fcfec3f2","Type":"ContainerStarted","Data":"333bde58c2e412b672c4f0683a8577ee909d6d7e1ef6d096de62bd74cf9ab2de"}
Sep 16 18:42:12.142502 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:12.142349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fww7m" event={"ID":"bbd239e1-4420-4b27-bf19-2e29fcfec3f2","Type":"ContainerStarted","Data":"b45bdc38dcea25ce71aae346f65155b3e3ef761189815191368a9ef2fc66ad9a"}
Sep 16 18:42:12.143465 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:12.143445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jcr5s" event={"ID":"7761e9ff-e826-4316-9a1a-876df7e23311","Type":"ContainerStarted","Data":"a80424b4a74d5dc6e708011152e6acdb0875ac5c7788cdaaeb1df7162d8b62a0"}
Sep 16 18:42:12.159923 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:12.159870    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fww7m" podStartSLOduration=2.159858685 podStartE2EDuration="2.159858685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:12.159043918 +0000 UTC m=+105.041867001" watchObservedRunningTime="2026-09-16 18:42:12.159858685 +0000 UTC m=+105.042681779"
Sep 16 18:42:14.150919 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.150878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jcr5s" event={"ID":"7761e9ff-e826-4316-9a1a-876df7e23311","Type":"ContainerStarted","Data":"b07796a872379682efaa1df243947bb571f677d85a94c474a860e8ca0e7dfb93"}
Sep 16 18:42:14.170535 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.170471    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-jcr5s" podStartSLOduration=2.281231253 podStartE2EDuration="4.170455047s" totalImagesPullingTime="1.889223794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:14.169979796 +0000 UTC m=+107.052802876" watchObservedRunningTime="2026-09-16 18:42:14.170455047 +0000 UTC m=+107.053278129"
Sep 16 18:42:14.587867 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.587836    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-njs2v"]
Sep 16 18:42:14.591051 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.591032    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.593618 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.593594    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-99dld\""
Sep 16 18:42:14.593838 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.593820    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:42:14.593902 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.593847    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:42:14.606431 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.606408    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-njs2v"]
Sep 16 18:42:14.689053 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.689026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/293464f7-a7bc-4529-8a46-a3266ec67b0a-crio-socket\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.689170 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.689056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/293464f7-a7bc-4529-8a46-a3266ec67b0a-data-volume\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.689170 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.689125    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.689170 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.689154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qp46\" (UniqueName: \"kubernetes.io/projected/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-api-access-6qp46\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.689285 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.689205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.789846 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.789823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/293464f7-a7bc-4529-8a46-a3266ec67b0a-crio-socket\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.789946 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.789848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/293464f7-a7bc-4529-8a46-a3266ec67b0a-data-volume\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.789946 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.789887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.789946 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.789904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qp46\" (UniqueName: \"kubernetes.io/projected/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-api-access-6qp46\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.789946 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.789940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.790151 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.789943    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/293464f7-a7bc-4529-8a46-a3266ec67b0a-crio-socket\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.790151 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:14.789975    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:14.790151 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:14.790033    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm podName:293464f7-a7bc-4529-8a46-a3266ec67b0a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:15.290014934 +0000 UTC m=+108.172837993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-njs2v" (UID: "293464f7-a7bc-4529-8a46-a3266ec67b0a") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:14.790151 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:14.790055    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:14.790151 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:14.790112    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls podName:293464f7-a7bc-4529-8a46-a3266ec67b0a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:15.290097643 +0000 UTC m=+108.172920719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-njs2v" (UID: "293464f7-a7bc-4529-8a46-a3266ec67b0a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:14.790336 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.790193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/293464f7-a7bc-4529-8a46-a3266ec67b0a-data-volume\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:14.801971 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:14.801948    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qp46\" (UniqueName: \"kubernetes.io/projected/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-api-access-6qp46\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:15.292122 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:15.292088    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:15.292474 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:15.292133    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:15.292474 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:15.292192    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:15.292474 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:15.292216    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:15.292474 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:15.292257    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm podName:293464f7-a7bc-4529-8a46-a3266ec67b0a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:16.292240541 +0000 UTC m=+109.175063599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-njs2v" (UID: "293464f7-a7bc-4529-8a46-a3266ec67b0a") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:15.292474 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:15.292271    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls podName:293464f7-a7bc-4529-8a46-a3266ec67b0a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:16.292265036 +0000 UTC m=+109.175088095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-njs2v" (UID: "293464f7-a7bc-4529-8a46-a3266ec67b0a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:16.299320 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:16.299276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:16.299320 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:16.299329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:16.299832 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:16.299416    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:16.299832 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:16.299432    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:16.299832 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:16.299489    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls podName:293464f7-a7bc-4529-8a46-a3266ec67b0a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:18.299475526 +0000 UTC m=+111.182298586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-njs2v" (UID: "293464f7-a7bc-4529-8a46-a3266ec67b0a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:16.299832 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:16.299503    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm podName:293464f7-a7bc-4529-8a46-a3266ec67b0a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:18.299496479 +0000 UTC m=+111.182319538 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-njs2v" (UID: "293464f7-a7bc-4529-8a46-a3266ec67b0a") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:16.513909 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:16.513881    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gt9pt_ae2b5270-597d-4a80-8f88-908422bec8a9/dns-node-resolver/0.log"
Sep 16 18:42:17.312184 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:17.312154    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9tntl_c3e6df42-134d-4588-bb47-5163ecb7925b/node-ca/0.log"
Sep 16 18:42:18.060219 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.060185    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-chn6r"]
Sep 16 18:42:18.063023 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.063002    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-chn6r"
Sep 16 18:42:18.065607 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.065579    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 18:42:18.065729 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.065615    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 18:42:18.066318 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.066299    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-p9khf\""
Sep 16 18:42:18.066417 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.066351    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 18:42:18.066417 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.066352    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 18:42:18.075685 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.075662    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-chn6r"]
Sep 16 18:42:18.109663 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.109641    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5nt49_f9947aa6-6269-453c-9a91-e95f47fffac3/serve-healthcheck-canary/0.log"
Sep 16 18:42:18.111809 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.111790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5x82\" (UniqueName: \"kubernetes.io/projected/93603594-beba-454e-b1dc-93ffa4a10641-kube-api-access-f5x82\") pod \"service-ca-568869fdc5-chn6r\" (UID: \"93603594-beba-454e-b1dc-93ffa4a10641\") " pod="openshift-service-ca/service-ca-568869fdc5-chn6r"
Sep 16 18:42:18.111901 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.111835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/93603594-beba-454e-b1dc-93ffa4a10641-signing-cabundle\") pod \"service-ca-568869fdc5-chn6r\" (UID: \"93603594-beba-454e-b1dc-93ffa4a10641\") " pod="openshift-service-ca/service-ca-568869fdc5-chn6r"
Sep 16 18:42:18.111901 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.111853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/93603594-beba-454e-b1dc-93ffa4a10641-signing-key\") pod \"service-ca-568869fdc5-chn6r\" (UID: \"93603594-beba-454e-b1dc-93ffa4a10641\") " pod="openshift-service-ca/service-ca-568869fdc5-chn6r"
Sep 16 18:42:18.212873 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.212854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5x82\" (UniqueName: \"kubernetes.io/projected/93603594-beba-454e-b1dc-93ffa4a10641-kube-api-access-f5x82\") pod \"service-ca-568869fdc5-chn6r\" (UID: \"93603594-beba-454e-b1dc-93ffa4a10641\") " pod="openshift-service-ca/service-ca-568869fdc5-chn6r"
Sep 16 18:42:18.212971 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.212900    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/93603594-beba-454e-b1dc-93ffa4a10641-signing-cabundle\") pod \"service-ca-568869fdc5-chn6r\" (UID: \"93603594-beba-454e-b1dc-93ffa4a10641\") " pod="openshift-service-ca/service-ca-568869fdc5-chn6r"
Sep 16 18:42:18.212971 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.212918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/93603594-beba-454e-b1dc-93ffa4a10641-signing-key\") pod \"service-ca-568869fdc5-chn6r\" (UID: \"93603594-beba-454e-b1dc-93ffa4a10641\") " pod="openshift-service-ca/service-ca-568869fdc5-chn6r"
Sep 16 18:42:18.213486 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.213466    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/93603594-beba-454e-b1dc-93ffa4a10641-signing-cabundle\") pod \"service-ca-568869fdc5-chn6r\" (UID: \"93603594-beba-454e-b1dc-93ffa4a10641\") " pod="openshift-service-ca/service-ca-568869fdc5-chn6r"
Sep 16 18:42:18.215423 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.215397    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/93603594-beba-454e-b1dc-93ffa4a10641-signing-key\") pod \"service-ca-568869fdc5-chn6r\" (UID: \"93603594-beba-454e-b1dc-93ffa4a10641\") " pod="openshift-service-ca/service-ca-568869fdc5-chn6r"
Sep 16 18:42:18.222119 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.222098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5x82\" (UniqueName: \"kubernetes.io/projected/93603594-beba-454e-b1dc-93ffa4a10641-kube-api-access-f5x82\") pod \"service-ca-568869fdc5-chn6r\" (UID: \"93603594-beba-454e-b1dc-93ffa4a10641\") " pod="openshift-service-ca/service-ca-568869fdc5-chn6r"
Sep 16 18:42:18.313675 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.313623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:18.313952 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.313689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:18.313952 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:18.313768    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.313952 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:18.313786    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:18.313952 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:18.313834    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm podName:293464f7-a7bc-4529-8a46-a3266ec67b0a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:22.313819409 +0000 UTC m=+115.196642493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-njs2v" (UID: "293464f7-a7bc-4529-8a46-a3266ec67b0a") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:18.313952 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:18.313849    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls podName:293464f7-a7bc-4529-8a46-a3266ec67b0a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:22.313842785 +0000 UTC m=+115.196665844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-njs2v" (UID: "293464f7-a7bc-4529-8a46-a3266ec67b0a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:18.372267 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.372245    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-chn6r"
Sep 16 18:42:18.643952 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:18.643890    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-chn6r"]
Sep 16 18:42:18.647111 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:18.647083    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93603594_beba_454e_b1dc_93ffa4a10641.slice/crio-141ef32bf4647454761122c10970a3ecb1ffa1ecd7e69cbe0ffd9c0d0cbe002e WatchSource:0}: Error finding container 141ef32bf4647454761122c10970a3ecb1ffa1ecd7e69cbe0ffd9c0d0cbe002e: Status 404 returned error can't find the container with id 141ef32bf4647454761122c10970a3ecb1ffa1ecd7e69cbe0ffd9c0d0cbe002e
Sep 16 18:42:19.164466 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:19.164427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-chn6r" event={"ID":"93603594-beba-454e-b1dc-93ffa4a10641","Type":"ContainerStarted","Data":"141ef32bf4647454761122c10970a3ecb1ffa1ecd7e69cbe0ffd9c0d0cbe002e"}
Sep 16 18:42:21.173369 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:21.173336    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-chn6r" event={"ID":"93603594-beba-454e-b1dc-93ffa4a10641","Type":"ContainerStarted","Data":"5397870702fc3925d756cb6d9da889db75784f0dd4e8846434b9124e6b1c3822"}
Sep 16 18:42:21.189832 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:21.189786    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-chn6r" podStartSLOduration=1.58833885 podStartE2EDuration="3.1897708s" totalImagesPullingTime="1.60143195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:21.189295104 +0000 UTC m=+114.072118185" watchObservedRunningTime="2026-09-16 18:42:21.1897708 +0000 UTC m=+114.072593881"
Sep 16 18:42:22.346384 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:22.346344    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:22.346831 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:22.346419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:22.346831 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:22.346419    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:42:22.346831 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:22.346488    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:22.346831 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:22.346528    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm podName:293464f7-a7bc-4529-8a46-a3266ec67b0a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:30.346509536 +0000 UTC m=+123.229332610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-njs2v" (UID: "293464f7-a7bc-4529-8a46-a3266ec67b0a") : configmap "kube-rbac-proxy" not found
Sep 16 18:42:22.346831 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:22.346565    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls podName:293464f7-a7bc-4529-8a46-a3266ec67b0a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:30.346547094 +0000 UTC m=+123.229370170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-njs2v" (UID: "293464f7-a7bc-4529-8a46-a3266ec67b0a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:30.404431 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:30.404388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:30.404923 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:30.404508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:30.404923 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:30.404520    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:30.404923 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:30.404588    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls podName:293464f7-a7bc-4529-8a46-a3266ec67b0a nodeName:}" failed. No retries permitted until 2026-09-16 18:42:46.404571949 +0000 UTC m=+139.287395008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-njs2v" (UID: "293464f7-a7bc-4529-8a46-a3266ec67b0a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:42:30.405141 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:30.405112    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/293464f7-a7bc-4529-8a46-a3266ec67b0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:37.359507 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:37.359447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:42:37.361932 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:37.361912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13bef125-8a64-4554-9ef9-fb4ddabc0628-metrics-certs\") pod \"network-metrics-daemon-n9vzt\" (UID: \"13bef125-8a64-4554-9ef9-fb4ddabc0628\") " pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:42:37.466781 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:37.466734    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx4q5\""
Sep 16 18:42:37.474101 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:37.474083    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vzt"
Sep 16 18:42:37.743103 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:37.743074    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n9vzt"]
Sep 16 18:42:37.745875 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:37.745848    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bef125_8a64_4554_9ef9_fb4ddabc0628.slice/crio-ff8ea603026b2d6a5c33516a6391f3f71db8aa3d787f3712a8edeb560931571e WatchSource:0}: Error finding container ff8ea603026b2d6a5c33516a6391f3f71db8aa3d787f3712a8edeb560931571e: Status 404 returned error can't find the container with id ff8ea603026b2d6a5c33516a6391f3f71db8aa3d787f3712a8edeb560931571e
Sep 16 18:42:38.228278 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:38.228236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9vzt" event={"ID":"13bef125-8a64-4554-9ef9-fb4ddabc0628","Type":"ContainerStarted","Data":"ff8ea603026b2d6a5c33516a6391f3f71db8aa3d787f3712a8edeb560931571e"}
Sep 16 18:42:39.233948 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:39.233906    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9vzt" event={"ID":"13bef125-8a64-4554-9ef9-fb4ddabc0628","Type":"ContainerStarted","Data":"5b0c63ae01e9f39ec17609b90f698b41b74a62f05625463352827c5955c89951"}
Sep 16 18:42:39.234428 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:39.233969    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9vzt" event={"ID":"13bef125-8a64-4554-9ef9-fb4ddabc0628","Type":"ContainerStarted","Data":"5bf42041397ec377787e93179bb19572d33d6f5b4a897b03a8629c098444e207"}
Sep 16 18:42:39.256407 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:39.256364    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n9vzt" podStartSLOduration=130.328176453 podStartE2EDuration="2m11.256350587s" totalImagesPullingTime="928.174134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:39.255615114 +0000 UTC m=+132.138438196" watchObservedRunningTime="2026-09-16 18:42:39.256350587 +0000 UTC m=+132.139173667"
Sep 16 18:42:43.613905 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.613862    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq"]
Sep 16 18:42:43.617075 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.617057    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq"
Sep 16 18:42:43.619787 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.619760    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:42:43.620152 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.620135    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xcdr5\""
Sep 16 18:42:43.620579 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.620563    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:42:43.621135 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.621113    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:42:43.621272 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.621118    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:42:43.626157 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.626130    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq"]
Sep 16 18:42:43.705454 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.705429    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqvpb\" (UniqueName: \"kubernetes.io/projected/34baf329-436b-44de-b066-0722bb1e11f1-kube-api-access-sqvpb\") pod \"managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq\" (UID: \"34baf329-436b-44de-b066-0722bb1e11f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq"
Sep 16 18:42:43.705553 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.705456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34baf329-436b-44de-b066-0722bb1e11f1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq\" (UID: \"34baf329-436b-44de-b066-0722bb1e11f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq"
Sep 16 18:42:43.805857 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.805832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqvpb\" (UniqueName: \"kubernetes.io/projected/34baf329-436b-44de-b066-0722bb1e11f1-kube-api-access-sqvpb\") pod \"managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq\" (UID: \"34baf329-436b-44de-b066-0722bb1e11f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq"
Sep 16 18:42:43.805959 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.805865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34baf329-436b-44de-b066-0722bb1e11f1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq\" (UID: \"34baf329-436b-44de-b066-0722bb1e11f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq"
Sep 16 18:42:43.808482 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.808459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/34baf329-436b-44de-b066-0722bb1e11f1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq\" (UID: \"34baf329-436b-44de-b066-0722bb1e11f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq"
Sep 16 18:42:43.819629 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.819604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqvpb\" (UniqueName: \"kubernetes.io/projected/34baf329-436b-44de-b066-0722bb1e11f1-kube-api-access-sqvpb\") pod \"managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq\" (UID: \"34baf329-436b-44de-b066-0722bb1e11f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq"
Sep 16 18:42:43.934345 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:43.934325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq"
Sep 16 18:42:44.201501 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:44.201423    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq"]
Sep 16 18:42:44.205129 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:44.205104    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34baf329_436b_44de_b066_0722bb1e11f1.slice/crio-6baf88b48a7b1fbb4112051742bc2ca58d6e8cbc96f4854bb0e3b75d12de57fb WatchSource:0}: Error finding container 6baf88b48a7b1fbb4112051742bc2ca58d6e8cbc96f4854bb0e3b75d12de57fb: Status 404 returned error can't find the container with id 6baf88b48a7b1fbb4112051742bc2ca58d6e8cbc96f4854bb0e3b75d12de57fb
Sep 16 18:42:44.251134 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:44.251102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq" event={"ID":"34baf329-436b-44de-b066-0722bb1e11f1","Type":"ContainerStarted","Data":"6baf88b48a7b1fbb4112051742bc2ca58d6e8cbc96f4854bb0e3b75d12de57fb"}
Sep 16 18:42:45.630190 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:45.630159    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d"]
Sep 16 18:42:45.638466 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:45.638372    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d"
Sep 16 18:42:45.641161 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:45.641105    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:42:45.641302 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:45.641156    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hxfpj\""
Sep 16 18:42:45.643162 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:45.643140    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d"]
Sep 16 18:42:45.721777 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:45.721734    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/04989fa3-1e2a-40cc-9624-3c061a9e88a5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qv69d\" (UID: \"04989fa3-1e2a-40cc-9624-3c061a9e88a5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d"
Sep 16 18:42:45.823242 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:45.823144    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/04989fa3-1e2a-40cc-9624-3c061a9e88a5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qv69d\" (UID: \"04989fa3-1e2a-40cc-9624-3c061a9e88a5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d"
Sep 16 18:42:45.825973 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:45.825945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/04989fa3-1e2a-40cc-9624-3c061a9e88a5-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qv69d\" (UID: \"04989fa3-1e2a-40cc-9624-3c061a9e88a5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d"
Sep 16 18:42:45.955633 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:45.955604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d"
Sep 16 18:42:46.428483 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:46.428456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:46.430597 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:46.430580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/293464f7-a7bc-4529-8a46-a3266ec67b0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njs2v\" (UID: \"293464f7-a7bc-4529-8a46-a3266ec67b0a\") " pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:46.599353 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:46.599272    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d"]
Sep 16 18:42:46.601901 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:46.601876    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04989fa3_1e2a_40cc_9624_3c061a9e88a5.slice/crio-c843bc40550fb9421bce52c9127c3d2e0f26e147ebd4e1a9fe3eebe1bf22380e WatchSource:0}: Error finding container c843bc40550fb9421bce52c9127c3d2e0f26e147ebd4e1a9fe3eebe1bf22380e: Status 404 returned error can't find the container with id c843bc40550fb9421bce52c9127c3d2e0f26e147ebd4e1a9fe3eebe1bf22380e
Sep 16 18:42:46.702600 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:46.702568    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-99dld\""
Sep 16 18:42:46.710288 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:46.710273    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-njs2v"
Sep 16 18:42:46.980280 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:46.980201    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-njs2v"]
Sep 16 18:42:46.984449 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:46.984420    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod293464f7_a7bc_4529_8a46_a3266ec67b0a.slice/crio-9ab605640c3ffcbd6e3fa239eece4644779178c4c37a7d9abb8290263327d68f WatchSource:0}: Error finding container 9ab605640c3ffcbd6e3fa239eece4644779178c4c37a7d9abb8290263327d68f: Status 404 returned error can't find the container with id 9ab605640c3ffcbd6e3fa239eece4644779178c4c37a7d9abb8290263327d68f
Sep 16 18:42:47.265728 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:47.265647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njs2v" event={"ID":"293464f7-a7bc-4529-8a46-a3266ec67b0a","Type":"ContainerStarted","Data":"fb5b0cded69d037d42fe2b70027a2c161be1fbe34a87f457595b023c548d9558"}
Sep 16 18:42:47.265728 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:47.265715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njs2v" event={"ID":"293464f7-a7bc-4529-8a46-a3266ec67b0a","Type":"ContainerStarted","Data":"9ab605640c3ffcbd6e3fa239eece4644779178c4c37a7d9abb8290263327d68f"}
Sep 16 18:42:47.267150 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:47.267111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d" event={"ID":"04989fa3-1e2a-40cc-9624-3c061a9e88a5","Type":"ContainerStarted","Data":"c843bc40550fb9421bce52c9127c3d2e0f26e147ebd4e1a9fe3eebe1bf22380e"}
Sep 16 18:42:47.268794 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:47.268773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq" event={"ID":"34baf329-436b-44de-b066-0722bb1e11f1","Type":"ContainerStarted","Data":"74065bd04f9f81b9ce1ac8cc68a91eabd712fdbf07399451582ddb7ce2950c4a"}
Sep 16 18:42:47.285765 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:47.285507    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5cf8f765d8-tvlgq" podStartSLOduration=2.097203793 podStartE2EDuration="4.285490453s" totalImagesPullingTime="2.18828666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:47.285250344 +0000 UTC m=+140.168073426" watchObservedRunningTime="2026-09-16 18:42:47.285490453 +0000 UTC m=+140.168313535"
Sep 16 18:42:48.276814 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:48.276772    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njs2v" event={"ID":"293464f7-a7bc-4529-8a46-a3266ec67b0a","Type":"ContainerStarted","Data":"db0999e423860c160d1c04c3a3715805bfb5d88d7e2efad8a4b7dad6781a38b9"}
Sep 16 18:42:48.278841 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:48.278809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d" event={"ID":"04989fa3-1e2a-40cc-9624-3c061a9e88a5","Type":"ContainerStarted","Data":"3a979bc78fa4320ae79479ec6d8809a2f604ea29e770f2c3708986031e9a7c97"}
Sep 16 18:42:48.297781 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:48.297704    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d" podStartSLOduration=2.2819078839999998 podStartE2EDuration="3.297690491s" totalImagesPullingTime="1.015782607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:48.297299614 +0000 UTC m=+141.180122717" watchObservedRunningTime="2026-09-16 18:42:48.297690491 +0000 UTC m=+141.180513619"
Sep 16 18:42:49.284631 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.284596    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njs2v" event={"ID":"293464f7-a7bc-4529-8a46-a3266ec67b0a","Type":"ContainerStarted","Data":"99c6359cdc6e04660da77ac45f1dc52778e1a77e53cf991f2c1a52e4b1152974"}
Sep 16 18:42:49.284965 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.284839    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d"
Sep 16 18:42:49.289722 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.289700    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qv69d"
Sep 16 18:42:49.305351 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.305272    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-njs2v" podStartSLOduration=33.348864612 podStartE2EDuration="35.305258546s" totalImagesPullingTime="1.956393934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:49.303822293 +0000 UTC m=+142.186645374" watchObservedRunningTime="2026-09-16 18:42:49.305258546 +0000 UTC m=+142.188081626"
Sep 16 18:42:49.730837 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.730804    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"]
Sep 16 18:42:49.734343 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.734327    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.737091 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.737060    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 18:42:49.737211 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.737098    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:42:49.737321 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.737306    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zmrdr\""
Sep 16 18:42:49.737382 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.737334    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:42:49.737696 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.737679    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:42:49.738239 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.738219    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 18:42:49.746345 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.746323    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"]
Sep 16 18:42:49.856328 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.856307    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmd9x\" (UniqueName: \"kubernetes.io/projected/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-kube-api-access-tmd9x\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.856445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.856350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.856445 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.856378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.856561 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.856489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.957176 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.957137    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.957297 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.957211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.957297 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.957241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmd9x\" (UniqueName: \"kubernetes.io/projected/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-kube-api-access-tmd9x\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.957297 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.957268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.957929 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.957905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.959732 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.959711    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.959869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.959853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:49.966941 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:49.966921    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmd9x\" (UniqueName: \"kubernetes.io/projected/3da8c784-e626-469b-8d9e-b9c0e7d9e44a-kube-api-access-tmd9x\") pod \"prometheus-operator-68bcd7f987-pv6hl\" (UID: \"3da8c784-e626-469b-8d9e-b9c0e7d9e44a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:50.044143 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:50.044073    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"
Sep 16 18:42:50.313497 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:50.313425    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl"]
Sep 16 18:42:50.316580 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:50.316552    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da8c784_e626_469b_8d9e_b9c0e7d9e44a.slice/crio-95babe76bb5c3f078e7f1c1de2dff5b8999f8c6922752e6ae31260ac7b710d4d WatchSource:0}: Error finding container 95babe76bb5c3f078e7f1c1de2dff5b8999f8c6922752e6ae31260ac7b710d4d: Status 404 returned error can't find the container with id 95babe76bb5c3f078e7f1c1de2dff5b8999f8c6922752e6ae31260ac7b710d4d
Sep 16 18:42:51.293144 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:51.293106    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl" event={"ID":"3da8c784-e626-469b-8d9e-b9c0e7d9e44a","Type":"ContainerStarted","Data":"95babe76bb5c3f078e7f1c1de2dff5b8999f8c6922752e6ae31260ac7b710d4d"}
Sep 16 18:42:52.299507 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.299467    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl" event={"ID":"3da8c784-e626-469b-8d9e-b9c0e7d9e44a","Type":"ContainerStarted","Data":"5218192a7f39d8a1e55061f8a0ccd1e9acbf459c21983b22e372e85a13f89dd4"}
Sep 16 18:42:52.299966 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.299527    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl" event={"ID":"3da8c784-e626-469b-8d9e-b9c0e7d9e44a","Type":"ContainerStarted","Data":"9842b075d1a923a27ec43770ed22505d14746476c4f14bba1de1d55585734223"}
Sep 16 18:42:52.322776 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.319087    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-pv6hl" podStartSLOduration=2.133063505 podStartE2EDuration="3.319066465s" totalImagesPullingTime="1.18600296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:52.317019384 +0000 UTC m=+145.199842465" watchObservedRunningTime="2026-09-16 18:42:52.319066465 +0000 UTC m=+145.201889549"
Sep 16 18:42:52.543361 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.543334    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f4fcf8d74-hslq7"]
Sep 16 18:42:52.546578 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.546557    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.548988 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.548966    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:42:52.548988 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.548977    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:42:52.549173 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.548970    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:42:52.549729 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.549670    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:42:52.549729 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.549697    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:42:52.549729 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.549713    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:42:52.549961 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.549670    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hxfv9\""
Sep 16 18:42:52.549961 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.549776    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:42:52.556315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.556295    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f4fcf8d74-hslq7"]
Sep 16 18:42:52.678243 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.678216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hctkg\" (UniqueName: \"kubernetes.io/projected/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-kube-api-access-hctkg\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.678389 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.678254    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-serving-cert\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.678389 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.678338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-oauth-config\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.678389 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.678363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-oauth-serving-cert\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.678389 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.678386    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-service-ca\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.678534 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.678402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-config\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.779117 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.779082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hctkg\" (UniqueName: \"kubernetes.io/projected/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-kube-api-access-hctkg\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.779117 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.779123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-serving-cert\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.779341 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.779169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-oauth-config\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.779341 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.779188    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-oauth-serving-cert\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.779341 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.779207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-service-ca\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.779341 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.779221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-config\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.780009 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.779985    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-oauth-serving-cert\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.780009 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.780006    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-service-ca\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.780159 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.779982    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-config\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.781852 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.781824    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-serving-cert\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.781942 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.781869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-oauth-config\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.788098 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.788075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hctkg\" (UniqueName: \"kubernetes.io/projected/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-kube-api-access-hctkg\") pod \"console-7f4fcf8d74-hslq7\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") " pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:52.855862 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:52.855810    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:42:53.132468 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:53.132395    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f4fcf8d74-hslq7"]
Sep 16 18:42:53.136850 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:53.136824    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3756240_cc4f_4aa5_9e82_72e10cbf2a23.slice/crio-526d36adf471c0c25ee4e10072c01ed1596fc0d105c1d3a05eec8e83118b2689 WatchSource:0}: Error finding container 526d36adf471c0c25ee4e10072c01ed1596fc0d105c1d3a05eec8e83118b2689: Status 404 returned error can't find the container with id 526d36adf471c0c25ee4e10072c01ed1596fc0d105c1d3a05eec8e83118b2689
Sep 16 18:42:53.306245 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:53.306219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4fcf8d74-hslq7" event={"ID":"b3756240-cc4f-4aa5-9e82-72e10cbf2a23","Type":"ContainerStarted","Data":"526d36adf471c0c25ee4e10072c01ed1596fc0d105c1d3a05eec8e83118b2689"}
Sep 16 18:42:54.156489 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.156452    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"]
Sep 16 18:42:54.161503 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.161068    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.164009 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.163486    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:42:54.164009 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.163490    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fbdx4\""
Sep 16 18:42:54.165100 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.164177    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:42:54.165100 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.164366    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:42:54.176962 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.175608    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t62ml"]
Sep 16 18:42:54.181946 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.181924    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"]
Sep 16 18:42:54.182928 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.182078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.184903 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.184658    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5gflp\""
Sep 16 18:42:54.184903 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.184704    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:42:54.185047 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.184909    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:42:54.186973 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.186693    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.290349    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31c13446-60a1-4b20-a197-cb3f08fcf388-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.290387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7a1769b-c075-40ca-a814-6b09b1b7ac62-metrics-client-ca\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.290436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl5pp\" (UniqueName: \"kubernetes.io/projected/c7a1769b-c075-40ca-a814-6b09b1b7ac62-kube-api-access-jl5pp\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.290483    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q89r\" (UniqueName: \"kubernetes.io/projected/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-api-access-2q89r\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.290505    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7a1769b-c075-40ca-a814-6b09b1b7ac62-sys\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.290528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7a1769b-c075-40ca-a814-6b09b1b7ac62-root\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.290552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-accelerators-collector-config\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.290585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-textfile\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.290609    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.290634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31c13446-60a1-4b20-a197-cb3f08fcf388-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.291808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-tls\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.291861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.291887    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.291910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-wtmp\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.292127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.291965    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.392804 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.392690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.392804 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.392767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31c13446-60a1-4b20-a197-cb3f08fcf388-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.393762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7a1769b-c075-40ca-a814-6b09b1b7ac62-metrics-client-ca\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.393842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5pp\" (UniqueName: \"kubernetes.io/projected/c7a1769b-c075-40ca-a814-6b09b1b7ac62-kube-api-access-jl5pp\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.393894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q89r\" (UniqueName: \"kubernetes.io/projected/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-api-access-2q89r\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.393918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7a1769b-c075-40ca-a814-6b09b1b7ac62-sys\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.393940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7a1769b-c075-40ca-a814-6b09b1b7ac62-root\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.393966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-accelerators-collector-config\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.394005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-textfile\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.394031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.394057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31c13446-60a1-4b20-a197-cb3f08fcf388-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.394084    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-tls\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.394118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.394143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.394167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-wtmp\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.394341    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-wtmp\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.394676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/31c13446-60a1-4b20-a197-cb3f08fcf388-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.395690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.395180    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7a1769b-c075-40ca-a814-6b09b1b7ac62-metrics-client-ca\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.396637 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.396600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31c13446-60a1-4b20-a197-cb3f08fcf388-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.396773 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:54.396756    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:42:54.396845 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:54.396813    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-tls podName:c7a1769b-c075-40ca-a814-6b09b1b7ac62 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:54.89679647 +0000 UTC m=+147.779619536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-tls") pod "node-exporter-t62ml" (UID: "c7a1769b-c075-40ca-a814-6b09b1b7ac62") : secret "node-exporter-tls" not found
Sep 16 18:42:54.397023 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:54.397009    2583 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 16 18:42:54.397087 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:54.397051    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-tls podName:31c13446-60a1-4b20-a197-cb3f08fcf388 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:54.89703846 +0000 UTC m=+147.779861526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-wlrhq" (UID: "31c13446-60a1-4b20-a197-cb3f08fcf388") : secret "kube-state-metrics-tls" not found
Sep 16 18:42:54.397651 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.397627    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.397764 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.397721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7a1769b-c075-40ca-a814-6b09b1b7ac62-root\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.397835 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.397789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7a1769b-c075-40ca-a814-6b09b1b7ac62-sys\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.398292 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.398194    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-accelerators-collector-config\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.398292 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.398241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-textfile\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.399418 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.399395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.411117 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.411063    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q89r\" (UniqueName: \"kubernetes.io/projected/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-api-access-2q89r\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.422145 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.422121    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.434645 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.434618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl5pp\" (UniqueName: \"kubernetes.io/projected/c7a1769b-c075-40ca-a814-6b09b1b7ac62-kube-api-access-jl5pp\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.898678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.898476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-tls\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:54.898678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.898534    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:54.898980 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:54.898949    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:42:54.899104 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:54.899035    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-tls podName:c7a1769b-c075-40ca-a814-6b09b1b7ac62 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:55.899011669 +0000 UTC m=+148.781834733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-tls") pod "node-exporter-t62ml" (UID: "c7a1769b-c075-40ca-a814-6b09b1b7ac62") : secret "node-exporter-tls" not found
Sep 16 18:42:54.901349 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:54.901329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/31c13446-60a1-4b20-a197-cb3f08fcf388-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-wlrhq\" (UID: \"31c13446-60a1-4b20-a197-cb3f08fcf388\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:55.077963 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:55.077926    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"
Sep 16 18:42:55.907492 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:55.907451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-tls\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:55.910214 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:55.910192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7a1769b-c075-40ca-a814-6b09b1b7ac62-node-exporter-tls\") pod \"node-exporter-t62ml\" (UID: \"c7a1769b-c075-40ca-a814-6b09b1b7ac62\") " pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:55.995893 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:55.995861    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t62ml"
Sep 16 18:42:56.010780 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:56.010731    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a1769b_c075_40ca_a814_6b09b1b7ac62.slice/crio-a397b427ed3d2d69d2ca0ac1b9648bea94fc84f6020b7d461cedbb13dcf11653 WatchSource:0}: Error finding container a397b427ed3d2d69d2ca0ac1b9648bea94fc84f6020b7d461cedbb13dcf11653: Status 404 returned error can't find the container with id a397b427ed3d2d69d2ca0ac1b9648bea94fc84f6020b7d461cedbb13dcf11653
Sep 16 18:42:56.237108 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:56.237075    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq"]
Sep 16 18:42:56.241346 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:56.241324    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c13446_60a1_4b20_a197_cb3f08fcf388.slice/crio-217ca609b1df7e9428d2721860b162c450df01a46f1c131c622d32fbbd1e46bb WatchSource:0}: Error finding container 217ca609b1df7e9428d2721860b162c450df01a46f1c131c622d32fbbd1e46bb: Status 404 returned error can't find the container with id 217ca609b1df7e9428d2721860b162c450df01a46f1c131c622d32fbbd1e46bb
Sep 16 18:42:56.317481 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:56.317456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t62ml" event={"ID":"c7a1769b-c075-40ca-a814-6b09b1b7ac62","Type":"ContainerStarted","Data":"a397b427ed3d2d69d2ca0ac1b9648bea94fc84f6020b7d461cedbb13dcf11653"}
Sep 16 18:42:56.318972 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:56.318946    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4fcf8d74-hslq7" event={"ID":"b3756240-cc4f-4aa5-9e82-72e10cbf2a23","Type":"ContainerStarted","Data":"e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6"}
Sep 16 18:42:56.320159 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:56.320140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq" event={"ID":"31c13446-60a1-4b20-a197-cb3f08fcf388","Type":"ContainerStarted","Data":"217ca609b1df7e9428d2721860b162c450df01a46f1c131c622d32fbbd1e46bb"}
Sep 16 18:42:56.338465 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:56.338425    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f4fcf8d74-hslq7" podStartSLOduration=1.465949631 podStartE2EDuration="4.33841243s" totalImagesPullingTime="2.872462799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:56.337686509 +0000 UTC m=+149.220509590" watchObservedRunningTime="2026-09-16 18:42:56.33841243 +0000 UTC m=+149.221235511"
Sep 16 18:42:57.326014 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:57.325983    2583 generic.go:356] "Generic (PLEG): container finished" podID="c7a1769b-c075-40ca-a814-6b09b1b7ac62" containerID="0bb46b340cd49147bc3b68daf3fbb98f9d7523fd1f11256ceac91e778c3c0eed" exitCode=0
Sep 16 18:42:57.326609 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:57.326559    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t62ml" event={"ID":"c7a1769b-c075-40ca-a814-6b09b1b7ac62","Type":"ContainerDied","Data":"0bb46b340cd49147bc3b68daf3fbb98f9d7523fd1f11256ceac91e778c3c0eed"}
Sep 16 18:42:58.336960 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.336921    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t62ml" event={"ID":"c7a1769b-c075-40ca-a814-6b09b1b7ac62","Type":"ContainerStarted","Data":"88ac91cfc3cb86de0590255ae6982d3f7fd02a2ad20b7d3c7f88a4cc37e2170f"}
Sep 16 18:42:58.337386 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.337022    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t62ml" event={"ID":"c7a1769b-c075-40ca-a814-6b09b1b7ac62","Type":"ContainerStarted","Data":"3ec614af70406305870a744ac340e8005c65f38a27c706d67a36e6f931d67810"}
Sep 16 18:42:58.339130 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.339105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq" event={"ID":"31c13446-60a1-4b20-a197-cb3f08fcf388","Type":"ContainerStarted","Data":"244e95858e9a42faad5cc7903d02d1efe715becb062bd092d612af34e3299b22"}
Sep 16 18:42:58.339263 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.339163    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq" event={"ID":"31c13446-60a1-4b20-a197-cb3f08fcf388","Type":"ContainerStarted","Data":"bdec5ce9978beaa01c2126269976480d47142ca0468f36cf3d38c54c6db2eed3"}
Sep 16 18:42:58.339263 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.339191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq" event={"ID":"31c13446-60a1-4b20-a197-cb3f08fcf388","Type":"ContainerStarted","Data":"cd654b9e070b85cdd8dbf266d5ff8c0d0e637a31d0babf57ae8f475fe52cfa8e"}
Sep 16 18:42:58.360711 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.360665    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t62ml" podStartSLOduration=3.41367591 podStartE2EDuration="4.360653131s" totalImagesPullingTime="868.050049ms" totalInitContainerRuntime="78.927172ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:58.359824581 +0000 UTC m=+151.242647661" watchObservedRunningTime="2026-09-16 18:42:58.360653131 +0000 UTC m=+151.243476282"
Sep 16 18:42:58.388558 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.388518    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wlrhq" podStartSLOduration=2.964939966 podStartE2EDuration="4.388503448s" totalImagesPullingTime="1.423563482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:58.387948747 +0000 UTC m=+151.270771830" watchObservedRunningTime="2026-09-16 18:42:58.388503448 +0000 UTC m=+151.271326528"
Sep 16 18:42:58.494696 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.494667    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64bf7496bf-767ls"]
Sep 16 18:42:58.497734 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.497720    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.500267 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.500240    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:42:58.500359 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.500275    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vsx7s\""
Sep 16 18:42:58.500424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.500356    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:42:58.500424 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.500284    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:42:58.500608 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.500596    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5jt6uqpkub715\""
Sep 16 18:42:58.500608 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.500602    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:42:58.530949 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.530924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/63654fe1-e483-4ce6-accb-662e776408e6-secret-metrics-server-client-certs\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.531044 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.530954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/63654fe1-e483-4ce6-accb-662e776408e6-metrics-server-audit-profiles\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.531044 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.530984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63654fe1-e483-4ce6-accb-662e776408e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.531162 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.531056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/63654fe1-e483-4ce6-accb-662e776408e6-client-ca-bundle\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.531162 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.531094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/63654fe1-e483-4ce6-accb-662e776408e6-secret-metrics-server-tls\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.531162 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.531131    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/63654fe1-e483-4ce6-accb-662e776408e6-audit-log\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.531295 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.531158    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64bf7496bf-767ls"]
Sep 16 18:42:58.531295 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.531217    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv64x\" (UniqueName: \"kubernetes.io/projected/63654fe1-e483-4ce6-accb-662e776408e6-kube-api-access-sv64x\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.631838 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.631773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/63654fe1-e483-4ce6-accb-662e776408e6-audit-log\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.631838 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.631805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv64x\" (UniqueName: \"kubernetes.io/projected/63654fe1-e483-4ce6-accb-662e776408e6-kube-api-access-sv64x\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.632020 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.631838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/63654fe1-e483-4ce6-accb-662e776408e6-secret-metrics-server-client-certs\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.632020 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.631868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/63654fe1-e483-4ce6-accb-662e776408e6-metrics-server-audit-profiles\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.632020 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.631905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63654fe1-e483-4ce6-accb-662e776408e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.632020 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.631955    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/63654fe1-e483-4ce6-accb-662e776408e6-client-ca-bundle\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.632020 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.632008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/63654fe1-e483-4ce6-accb-662e776408e6-secret-metrics-server-tls\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.632219 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.632206    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/63654fe1-e483-4ce6-accb-662e776408e6-audit-log\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.632676 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.632646    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63654fe1-e483-4ce6-accb-662e776408e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.632924 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.632850    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/63654fe1-e483-4ce6-accb-662e776408e6-metrics-server-audit-profiles\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.634482 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.634449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/63654fe1-e483-4ce6-accb-662e776408e6-secret-metrics-server-client-certs\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.634579 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.634569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/63654fe1-e483-4ce6-accb-662e776408e6-secret-metrics-server-tls\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.634639 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.634583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/63654fe1-e483-4ce6-accb-662e776408e6-client-ca-bundle\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.641315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.641294    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv64x\" (UniqueName: \"kubernetes.io/projected/63654fe1-e483-4ce6-accb-662e776408e6-kube-api-access-sv64x\") pod \"metrics-server-64bf7496bf-767ls\" (UID: \"63654fe1-e483-4ce6-accb-662e776408e6\") " pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.807414 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.807374    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:42:58.933374 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.933347    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"]
Sep 16 18:42:58.937512 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.937492    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"
Sep 16 18:42:58.942444 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.942425    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gq28p\""
Sep 16 18:42:58.942778 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.942760    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:42:58.952046 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:58.952027    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"]
Sep 16 18:42:59.034902 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.034879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgb5x\" (UniqueName: \"kubernetes.io/projected/d2e60300-e117-4a73-831f-6be65e3ac1c5-kube-api-access-zgb5x\") pod \"monitoring-plugin-5d85778b9d-cw54s\" (UID: \"d2e60300-e117-4a73-831f-6be65e3ac1c5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"
Sep 16 18:42:59.034981 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.034919    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2e60300-e117-4a73-831f-6be65e3ac1c5-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cw54s\" (UID: \"d2e60300-e117-4a73-831f-6be65e3ac1c5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"
Sep 16 18:42:59.107191 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.107169    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64bf7496bf-767ls"]
Sep 16 18:42:59.107858 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:42:59.107835    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63654fe1_e483_4ce6_accb_662e776408e6.slice/crio-9928f2f0fbcf3cdbe7f9aae0e2b16af82039c0eb403cdf5b347856354509a4d3 WatchSource:0}: Error finding container 9928f2f0fbcf3cdbe7f9aae0e2b16af82039c0eb403cdf5b347856354509a4d3: Status 404 returned error can't find the container with id 9928f2f0fbcf3cdbe7f9aae0e2b16af82039c0eb403cdf5b347856354509a4d3
Sep 16 18:42:59.135852 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.135828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgb5x\" (UniqueName: \"kubernetes.io/projected/d2e60300-e117-4a73-831f-6be65e3ac1c5-kube-api-access-zgb5x\") pod \"monitoring-plugin-5d85778b9d-cw54s\" (UID: \"d2e60300-e117-4a73-831f-6be65e3ac1c5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"
Sep 16 18:42:59.135959 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.135863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2e60300-e117-4a73-831f-6be65e3ac1c5-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cw54s\" (UID: \"d2e60300-e117-4a73-831f-6be65e3ac1c5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"
Sep 16 18:42:59.136013 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:59.135967    2583 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 18:42:59.136013 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:42:59.136013    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e60300-e117-4a73-831f-6be65e3ac1c5-monitoring-plugin-cert podName:d2e60300-e117-4a73-831f-6be65e3ac1c5 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:59.63599834 +0000 UTC m=+152.518821400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d2e60300-e117-4a73-831f-6be65e3ac1c5-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-cw54s" (UID: "d2e60300-e117-4a73-831f-6be65e3ac1c5") : secret "monitoring-plugin-cert" not found
Sep 16 18:42:59.156909 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.156885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgb5x\" (UniqueName: \"kubernetes.io/projected/d2e60300-e117-4a73-831f-6be65e3ac1c5-kube-api-access-zgb5x\") pod \"monitoring-plugin-5d85778b9d-cw54s\" (UID: \"d2e60300-e117-4a73-831f-6be65e3ac1c5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"
Sep 16 18:42:59.345168 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.345086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64bf7496bf-767ls" event={"ID":"63654fe1-e483-4ce6-accb-662e776408e6","Type":"ContainerStarted","Data":"9928f2f0fbcf3cdbe7f9aae0e2b16af82039c0eb403cdf5b347856354509a4d3"}
Sep 16 18:42:59.410888 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.410862    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"]
Sep 16 18:42:59.414390 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.414373    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.416965 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.416926    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:42:59.416965 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.416942    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:42:59.416965 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.416934    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pt5dl\""
Sep 16 18:42:59.417229 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.416974    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:42:59.417372 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.417356    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:42:59.417430 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.417414    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:42:59.423111 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.423091    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:42:59.437996 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.437972    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"]
Sep 16 18:42:59.439637 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.439542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e07a579f-0139-4090-a990-0537655d9ba5-serving-certs-ca-bundle\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.439637 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.439594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e07a579f-0139-4090-a990-0537655d9ba5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.439839 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.439693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-secret-telemeter-client\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.439839 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.439730    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-federate-client-tls\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.439839 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.439775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq7p4\" (UniqueName: \"kubernetes.io/projected/e07a579f-0139-4090-a990-0537655d9ba5-kube-api-access-lq7p4\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.440051 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.439941    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.440051 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.439970    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e07a579f-0139-4090-a990-0537655d9ba5-metrics-client-ca\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.440051 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.440000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-telemeter-client-tls\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.541359 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.541329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-secret-telemeter-client\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.541516 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.541374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-federate-client-tls\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.541516 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.541397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq7p4\" (UniqueName: \"kubernetes.io/projected/e07a579f-0139-4090-a990-0537655d9ba5-kube-api-access-lq7p4\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.541630 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.541601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.541682 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.541637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e07a579f-0139-4090-a990-0537655d9ba5-metrics-client-ca\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.541725 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.541670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-telemeter-client-tls\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.541785 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.541764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e07a579f-0139-4090-a990-0537655d9ba5-serving-certs-ca-bundle\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.541835 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.541808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e07a579f-0139-4090-a990-0537655d9ba5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.542490 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.542463    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e07a579f-0139-4090-a990-0537655d9ba5-metrics-client-ca\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.542707 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.542672    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e07a579f-0139-4090-a990-0537655d9ba5-serving-certs-ca-bundle\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.542828 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.542729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e07a579f-0139-4090-a990-0537655d9ba5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.544494 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.544473    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-federate-client-tls\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.544882 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.544860    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.545040 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.545022    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-secret-telemeter-client\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.545114 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.545098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e07a579f-0139-4090-a990-0537655d9ba5-telemeter-client-tls\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.562106 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.562081    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq7p4\" (UniqueName: \"kubernetes.io/projected/e07a579f-0139-4090-a990-0537655d9ba5-kube-api-access-lq7p4\") pod \"telemeter-client-6f9d8c4559-xzdtc\" (UID: \"e07a579f-0139-4090-a990-0537655d9ba5\") " pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.642804 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.642726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2e60300-e117-4a73-831f-6be65e3ac1c5-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cw54s\" (UID: \"d2e60300-e117-4a73-831f-6be65e3ac1c5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"
Sep 16 18:42:59.645323 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.645305    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2e60300-e117-4a73-831f-6be65e3ac1c5-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cw54s\" (UID: \"d2e60300-e117-4a73-831f-6be65e3ac1c5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"
Sep 16 18:42:59.725862 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.725837    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"
Sep 16 18:42:59.848724 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:42:59.848691    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"
Sep 16 18:43:00.019682 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.019651    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc"]
Sep 16 18:43:00.036199 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.036173    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f4fcf8d74-hslq7"]
Sep 16 18:43:00.103530 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.103499    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fcf944dc5-tlnjb"]
Sep 16 18:43:00.107014 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.106990    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.125476 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.125452    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:43:00.132635 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.132615    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fcf944dc5-tlnjb"]
Sep 16 18:43:00.147300 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.147270    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8vtd\" (UniqueName: \"kubernetes.io/projected/2f183772-94f0-4886-b14d-84981be01c8c-kube-api-access-g8vtd\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.147429 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.147343    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-serving-cert\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.147429 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.147391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-oauth-serving-cert\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.147538 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.147446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-trusted-ca-bundle\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.147538 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.147469    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-oauth-config\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.147538 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.147491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-console-config\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.147728 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.147559    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-service-ca\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.147889 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.147869    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"]
Sep 16 18:43:00.248522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.248487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8vtd\" (UniqueName: \"kubernetes.io/projected/2f183772-94f0-4886-b14d-84981be01c8c-kube-api-access-g8vtd\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.248707 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.248571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-serving-cert\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.248800 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.248708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-oauth-serving-cert\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.249548 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.249008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-trusted-ca-bundle\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.249548 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.249058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-oauth-config\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.249548 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.249080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-console-config\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.249548 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.249169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-service-ca\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.249859 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.249600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-oauth-serving-cert\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.249905 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.249851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-service-ca\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.250551 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.250522    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-trusted-ca-bundle\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.251858 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.251834    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-console-config\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.255001 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.252626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-serving-cert\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.255655 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.255631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-oauth-config\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.262843 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.262818    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8vtd\" (UniqueName: \"kubernetes.io/projected/2f183772-94f0-4886-b14d-84981be01c8c-kube-api-access-g8vtd\") pod \"console-7fcf944dc5-tlnjb\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") " pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.352537 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:43:00.352504    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07a579f_0139_4090_a990_0537655d9ba5.slice/crio-706aa71c7a6f093e635df8ecea055f64bb73faac0e5c85e842c5dcce57cc1d07 WatchSource:0}: Error finding container 706aa71c7a6f093e635df8ecea055f64bb73faac0e5c85e842c5dcce57cc1d07: Status 404 returned error can't find the container with id 706aa71c7a6f093e635df8ecea055f64bb73faac0e5c85e842c5dcce57cc1d07
Sep 16 18:43:00.353202 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:43:00.353127    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e60300_e117_4a73_831f_6be65e3ac1c5.slice/crio-f9f059323aa32b33490445eaf0ce1d8e492fa300a33e13c1ee54887faf6f800e WatchSource:0}: Error finding container f9f059323aa32b33490445eaf0ce1d8e492fa300a33e13c1ee54887faf6f800e: Status 404 returned error can't find the container with id f9f059323aa32b33490445eaf0ce1d8e492fa300a33e13c1ee54887faf6f800e
Sep 16 18:43:00.431315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.431262    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:00.523956 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.523883    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:00.528516 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.528491    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.536060 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.535850    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:43:00.536060 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.535892    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:43:00.536060 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.535850    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:43:00.536292 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.536150    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:43:00.537231 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.537212    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:43:00.537982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.537961    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:43:00.539020 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.538904    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rrj8v\""
Sep 16 18:43:00.539020 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.538909    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:43:00.540254 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.539393    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:43:00.540254 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.539607    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:43:00.540254 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.539840    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7akr39utptdeq\""
Sep 16 18:43:00.540254 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.539917    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:43:00.540254 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.540019    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:43:00.540254 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.540147    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:43:00.540549 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.540322    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:43:00.540549 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.540492    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:43:00.552132 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552226 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552144    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552226 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552226 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552190    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552226 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btqkh\" (UniqueName: \"kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-kube-api-access-btqkh\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552419 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552256    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552419 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552319    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552419 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552419 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552379    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552419 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552593 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552439    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552593 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552593 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552593 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552593 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552572    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552838 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552838 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552838 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.552838 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.553022 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.552856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.555541 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.555518    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:00.653282 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.653253    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.653430 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.653291    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.653430 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.653311    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.653430 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.653328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.653610 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.653461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.654205 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.654176    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.654342 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.654177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.654429 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.654337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.654429 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.654390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.654429 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.654421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655153    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.655982 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.655508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqkh\" (UniqueName: \"kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-kube-api-access-btqkh\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.656657 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.656478    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.656657 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.656600    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.657496 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.656972    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.657496 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.656971    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.657496 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.657178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.657496 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.657291    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.657787 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.657577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.657945 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.657920    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.658212 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.658145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.658548 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.658523    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.659358 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.659335    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config-out\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.659772 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.659721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.659879 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.659767    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.660426 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.660406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-web-config\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.660862 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.660842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.661125 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.661084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.661325 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.661307    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.670275 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.670257    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqkh\" (UniqueName: \"kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-kube-api-access-btqkh\") pod \"prometheus-k8s-0\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:00.723367 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.723340    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fcf944dc5-tlnjb"]
Sep 16 18:43:00.726561 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:43:00.726537    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f183772_94f0_4886_b14d_84981be01c8c.slice/crio-1188f1e4cbde0f6383c0fcb121d23d3263e7a5ba6156ccc63f8235570af0bfa7 WatchSource:0}: Error finding container 1188f1e4cbde0f6383c0fcb121d23d3263e7a5ba6156ccc63f8235570af0bfa7: Status 404 returned error can't find the container with id 1188f1e4cbde0f6383c0fcb121d23d3263e7a5ba6156ccc63f8235570af0bfa7
Sep 16 18:43:00.842573 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:00.842333    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:01.161544 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:01.160923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:43:01.165007 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:43:01.164972    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0e5be6_ca89_4cc3_b09d_47b6f745a936.slice/crio-54aa2bf191b147829e509231bc2da12ae3fec91ffe56d07bd8649ae4c011da6a WatchSource:0}: Error finding container 54aa2bf191b147829e509231bc2da12ae3fec91ffe56d07bd8649ae4c011da6a: Status 404 returned error can't find the container with id 54aa2bf191b147829e509231bc2da12ae3fec91ffe56d07bd8649ae4c011da6a
Sep 16 18:43:01.356245 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:01.356208    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcf944dc5-tlnjb" event={"ID":"2f183772-94f0-4886-b14d-84981be01c8c","Type":"ContainerStarted","Data":"39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7"}
Sep 16 18:43:01.356651 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:01.356303    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcf944dc5-tlnjb" event={"ID":"2f183772-94f0-4886-b14d-84981be01c8c","Type":"ContainerStarted","Data":"1188f1e4cbde0f6383c0fcb121d23d3263e7a5ba6156ccc63f8235570af0bfa7"}
Sep 16 18:43:01.359231 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:01.359174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64bf7496bf-767ls" event={"ID":"63654fe1-e483-4ce6-accb-662e776408e6","Type":"ContainerStarted","Data":"5c1983a61ca95f776a7cfd39289a8060c3fe1f759ec423bac2d18eb59ec39832"}
Sep 16 18:43:01.360996 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:01.360967    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerStarted","Data":"54aa2bf191b147829e509231bc2da12ae3fec91ffe56d07bd8649ae4c011da6a"}
Sep 16 18:43:01.362486 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:01.362427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s" event={"ID":"d2e60300-e117-4a73-831f-6be65e3ac1c5","Type":"ContainerStarted","Data":"f9f059323aa32b33490445eaf0ce1d8e492fa300a33e13c1ee54887faf6f800e"}
Sep 16 18:43:01.364197 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:01.364171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc" event={"ID":"e07a579f-0139-4090-a990-0537655d9ba5","Type":"ContainerStarted","Data":"706aa71c7a6f093e635df8ecea055f64bb73faac0e5c85e842c5dcce57cc1d07"}
Sep 16 18:43:01.400148 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:01.400099    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fcf944dc5-tlnjb" podStartSLOduration=1.40008186 podStartE2EDuration="1.40008186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:01.378560226 +0000 UTC m=+154.261383333" watchObservedRunningTime="2026-09-16 18:43:01.40008186 +0000 UTC m=+154.282904942"
Sep 16 18:43:01.401620 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:01.401582    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-64bf7496bf-767ls" podStartSLOduration=2.111806233 podStartE2EDuration="3.401570447s" totalImagesPullingTime="1.289764214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:01.399223996 +0000 UTC m=+154.282047078" watchObservedRunningTime="2026-09-16 18:43:01.401570447 +0000 UTC m=+154.284393527"
Sep 16 18:43:02.856448 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:02.856426    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:43:03.375756 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.375697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc" event={"ID":"e07a579f-0139-4090-a990-0537655d9ba5","Type":"ContainerStarted","Data":"3956b2effadd1e276d6d7b0d58a947adf9b765c93dde777dd68ca4211f8a7e11"}
Sep 16 18:43:03.376016 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.375793    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc" event={"ID":"e07a579f-0139-4090-a990-0537655d9ba5","Type":"ContainerStarted","Data":"b5a175db074db94bc6ec371945e7830c8b79e838741ecc971dbf729cbbf62e14"}
Sep 16 18:43:03.376016 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.375823    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc" event={"ID":"e07a579f-0139-4090-a990-0537655d9ba5","Type":"ContainerStarted","Data":"c05ca1c21e3fc1f5392df881df114c5125477616bd946ba4222427c2aa9c7cab"}
Sep 16 18:43:03.377458 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.377434    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerID="5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab" exitCode=0
Sep 16 18:43:03.377564 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.377517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerDied","Data":"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"}
Sep 16 18:43:03.379265 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.379240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s" event={"ID":"d2e60300-e117-4a73-831f-6be65e3ac1c5","Type":"ContainerStarted","Data":"1f73d2533f83902016a0801c4b0a3bc31db76860c72add2f3182c08979b718d6"}
Sep 16 18:43:03.379463 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.379441    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"
Sep 16 18:43:03.384925 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.384906    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s"
Sep 16 18:43:03.400327 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.400254    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f9d8c4559-xzdtc" podStartSLOduration=2.187199459 podStartE2EDuration="4.400235557s" totalImagesPullingTime="2.213036098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:03.399371978 +0000 UTC m=+156.282195062" watchObservedRunningTime="2026-09-16 18:43:03.400235557 +0000 UTC m=+156.283058641"
Sep 16 18:43:03.418538 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.418491    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cw54s" podStartSLOduration=3.211186303 podStartE2EDuration="5.418476318s" totalImagesPullingTime="2.207290015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:03.417658535 +0000 UTC m=+156.300481618" watchObservedRunningTime="2026-09-16 18:43:03.418476318 +0000 UTC m=+156.301299460"
Sep 16 18:43:03.564336 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.564306    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-674957ddc8-767mt"]
Sep 16 18:43:03.568357 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.568341    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.576856 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.576836    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-674957ddc8-767mt"]
Sep 16 18:43:03.687405 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.687381    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksbnb\" (UniqueName: \"kubernetes.io/projected/e85973ce-26d0-4619-b38d-7ba9cfc474f4-kube-api-access-ksbnb\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.687499 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.687412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-trusted-ca-bundle\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.687539 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.687503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-oauth-config\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.687539 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.687528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-service-ca\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.687612 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.687553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-config\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.687612 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.687585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-serving-cert\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.687612 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.687605    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-oauth-serving-cert\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.788764 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.788709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-oauth-config\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.788888 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.788766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-service-ca\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.788888 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.788804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-config\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.788888 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.788849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-serving-cert\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.789040 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.788974    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-oauth-serving-cert\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.789096 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.789042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksbnb\" (UniqueName: \"kubernetes.io/projected/e85973ce-26d0-4619-b38d-7ba9cfc474f4-kube-api-access-ksbnb\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.789096 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.789072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-trusted-ca-bundle\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.789678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.789614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-config\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.789678 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.789614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-service-ca\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.789861 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.789708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-oauth-serving-cert\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.789968 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.789945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-trusted-ca-bundle\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.791374 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.791347    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-oauth-config\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.791554 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.791535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-serving-cert\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.802256 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.802232    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksbnb\" (UniqueName: \"kubernetes.io/projected/e85973ce-26d0-4619-b38d-7ba9cfc474f4-kube-api-access-ksbnb\") pod \"console-674957ddc8-767mt\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") " pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:03.876845 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:03.876822    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:04.160921 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:04.160759    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-674957ddc8-767mt"]
Sep 16 18:43:04.163543 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:43:04.163511    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85973ce_26d0_4619_b38d_7ba9cfc474f4.slice/crio-cc3fda66dbe5bae0996287577b1565d0b17d7a65aabca86ae2e17afc9f7a0134 WatchSource:0}: Error finding container cc3fda66dbe5bae0996287577b1565d0b17d7a65aabca86ae2e17afc9f7a0134: Status 404 returned error can't find the container with id cc3fda66dbe5bae0996287577b1565d0b17d7a65aabca86ae2e17afc9f7a0134
Sep 16 18:43:04.390262 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:04.389618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674957ddc8-767mt" event={"ID":"e85973ce-26d0-4619-b38d-7ba9cfc474f4","Type":"ContainerStarted","Data":"4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6"}
Sep 16 18:43:04.390262 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:04.389668    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674957ddc8-767mt" event={"ID":"e85973ce-26d0-4619-b38d-7ba9cfc474f4","Type":"ContainerStarted","Data":"cc3fda66dbe5bae0996287577b1565d0b17d7a65aabca86ae2e17afc9f7a0134"}
Sep 16 18:43:04.410430 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:04.410363    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-674957ddc8-767mt" podStartSLOduration=1.4103442259999999 podStartE2EDuration="1.410344226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:04.40906441 +0000 UTC m=+157.291887494" watchObservedRunningTime="2026-09-16 18:43:04.410344226 +0000 UTC m=+157.293167309"
Sep 16 18:43:04.797157 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:43:04.797121    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5" podUID="87eb20c3-96fb-47c9-9b08-b4b4042a374b"
Sep 16 18:43:04.812368 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:43:04.812320    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2qbqd" podUID="aeb3b2b9-dea0-4346-9418-9b9394b13d82"
Sep 16 18:43:05.393341 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:05.393311    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:43:05.393341 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:05.393320    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:43:05.463866 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:05.463835    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb8d7c95b-djcg5"]
Sep 16 18:43:05.464217 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:43:05.464187    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5" podUID="87eb20c3-96fb-47c9-9b08-b4b4042a374b"
Sep 16 18:43:06.397401 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.397368    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:43:06.406619 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.406593    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:43:06.516082 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.516055    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7x75\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-kube-api-access-m7x75\") pod \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") "
Sep 16 18:43:06.516203 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.516095    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-certificates\") pod \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") "
Sep 16 18:43:06.516203 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.516163    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-trusted-ca\") pod \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") "
Sep 16 18:43:06.516320 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.516294    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-installation-pull-secrets\") pod \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") "
Sep 16 18:43:06.516375 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.516328    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-bound-sa-token\") pod \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") "
Sep 16 18:43:06.516430 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.516377    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-image-registry-private-configuration\") pod \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") "
Sep 16 18:43:06.516483 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.516458    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87eb20c3-96fb-47c9-9b08-b4b4042a374b-ca-trust-extracted\") pod \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\" (UID: \"87eb20c3-96fb-47c9-9b08-b4b4042a374b\") "
Sep 16 18:43:06.516574 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.516554    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "87eb20c3-96fb-47c9-9b08-b4b4042a374b" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:06.516640 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.516579    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "87eb20c3-96fb-47c9-9b08-b4b4042a374b" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:06.516855 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.516823    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87eb20c3-96fb-47c9-9b08-b4b4042a374b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "87eb20c3-96fb-47c9-9b08-b4b4042a374b" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:43:06.517184 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.517106    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87eb20c3-96fb-47c9-9b08-b4b4042a374b-ca-trust-extracted\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.517184 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.517130    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-certificates\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.517184 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.517147    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87eb20c3-96fb-47c9-9b08-b4b4042a374b-trusted-ca\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.518393 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.518369    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-kube-api-access-m7x75" (OuterVolumeSpecName: "kube-api-access-m7x75") pod "87eb20c3-96fb-47c9-9b08-b4b4042a374b" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b"). InnerVolumeSpecName "kube-api-access-m7x75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:06.518725 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.518693    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "87eb20c3-96fb-47c9-9b08-b4b4042a374b" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:06.519139 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.519113    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "87eb20c3-96fb-47c9-9b08-b4b4042a374b" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:06.519139 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.519120    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "87eb20c3-96fb-47c9-9b08-b4b4042a374b" (UID: "87eb20c3-96fb-47c9-9b08-b4b4042a374b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:06.618614 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.618585    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7x75\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-kube-api-access-m7x75\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.618614 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.618612    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-installation-pull-secrets\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.618799 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.618621    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-bound-sa-token\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:06.618799 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:06.618630    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87eb20c3-96fb-47c9-9b08-b4b4042a374b-image-registry-private-configuration\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:07.404788 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:07.404733    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb8d7c95b-djcg5"
Sep 16 18:43:07.404788 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:07.404775    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerStarted","Data":"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"}
Sep 16 18:43:07.405319 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:07.404896    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerStarted","Data":"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"}
Sep 16 18:43:07.451978 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:07.451950    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb8d7c95b-djcg5"]
Sep 16 18:43:07.454205 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:07.454177    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fb8d7c95b-djcg5"]
Sep 16 18:43:07.527060 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:07.527025    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87eb20c3-96fb-47c9-9b08-b4b4042a374b-registry-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:07.660283 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:07.660211    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87eb20c3-96fb-47c9-9b08-b4b4042a374b" path="/var/lib/kubelet/pods/87eb20c3-96fb-47c9-9b08-b4b4042a374b/volumes"
Sep 16 18:43:09.418530 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:09.418487    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerStarted","Data":"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"}
Sep 16 18:43:09.418983 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:09.418614    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerStarted","Data":"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"}
Sep 16 18:43:09.418983 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:09.418664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerStarted","Data":"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"}
Sep 16 18:43:09.418983 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:09.418702    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerStarted","Data":"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"}
Sep 16 18:43:09.456074 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:09.456026    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.052068948 podStartE2EDuration="9.4560101s" totalImagesPullingTime="6.373325921s" totalInitContainerRuntime="30.615231ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:09.454216979 +0000 UTC m=+162.337040061" watchObservedRunningTime="2026-09-16 18:43:09.4560101 +0000 UTC m=+162.338833181"
Sep 16 18:43:09.648003 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:09.647963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:43:09.650530 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:09.650507    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeb3b2b9-dea0-4346-9418-9b9394b13d82-metrics-tls\") pod \"dns-default-2qbqd\" (UID: \"aeb3b2b9-dea0-4346-9418-9b9394b13d82\") " pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:43:09.896520 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:09.896494    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zm88x\""
Sep 16 18:43:09.904379 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:09.904356    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:43:10.186120 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:10.186093    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2qbqd"]
Sep 16 18:43:10.188647 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:43:10.188625    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb3b2b9_dea0_4346_9418_9b9394b13d82.slice/crio-43cebf1c3723a7902a3ccbd92ae59b4ce99695c8508f6624ec9d96295329a6bb WatchSource:0}: Error finding container 43cebf1c3723a7902a3ccbd92ae59b4ce99695c8508f6624ec9d96295329a6bb: Status 404 returned error can't find the container with id 43cebf1c3723a7902a3ccbd92ae59b4ce99695c8508f6624ec9d96295329a6bb
Sep 16 18:43:10.423900 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:10.423869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2qbqd" event={"ID":"aeb3b2b9-dea0-4346-9418-9b9394b13d82","Type":"ContainerStarted","Data":"43cebf1c3723a7902a3ccbd92ae59b4ce99695c8508f6624ec9d96295329a6bb"}
Sep 16 18:43:10.431618 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:10.431589    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:10.431724 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:10.431638    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:10.436619 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:10.436565    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:10.842568 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:10.842495    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:43:11.433966 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:11.433937    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:12.434132 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:12.434098    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2qbqd" event={"ID":"aeb3b2b9-dea0-4346-9418-9b9394b13d82","Type":"ContainerStarted","Data":"cd00ea7d6d755f0794fb091c1d8788056b22a0e2cacbc313ddd8b5383b73e08b"}
Sep 16 18:43:12.434574 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:12.434183    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2qbqd" event={"ID":"aeb3b2b9-dea0-4346-9418-9b9394b13d82","Type":"ContainerStarted","Data":"46671d15dbface286ad9980e53e327e6a4a76d67e517af5a2c9fd482be41b084"}
Sep 16 18:43:12.434574 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:12.434381    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:43:12.452734 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:12.452552    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2qbqd" podStartSLOduration=129.720572473 podStartE2EDuration="2m11.452531608s" totalImagesPullingTime="1.731959135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:12.451769193 +0000 UTC m=+165.334592274" watchObservedRunningTime="2026-09-16 18:43:12.452531608 +0000 UTC m=+165.335354693"
Sep 16 18:43:13.877062 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:13.877029    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:13.877537 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:13.877129    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:13.881616 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:13.881595    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:14.445646 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:14.445619    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:43:14.510567 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:14.510531    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fcf944dc5-tlnjb"]
Sep 16 18:43:18.807807 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:18.807754    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:43:18.808234 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:18.807839    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:43:22.440586 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:22.440547    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2qbqd"
Sep 16 18:43:25.057847 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.057805    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f4fcf8d74-hslq7" podUID="b3756240-cc4f-4aa5-9e82-72e10cbf2a23" containerName="console" containerID="cri-o://e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6" gracePeriod=15
Sep 16 18:43:25.294631 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.294611    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:43:25.384176 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.384114    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-config\") pod \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") "
Sep 16 18:43:25.384176 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.384173    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-service-ca\") pod \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") "
Sep 16 18:43:25.384345 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.384252    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hctkg\" (UniqueName: \"kubernetes.io/projected/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-kube-api-access-hctkg\") pod \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") "
Sep 16 18:43:25.384345 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.384296    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-serving-cert\") pod \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") "
Sep 16 18:43:25.384345 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.384323    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-oauth-config\") pod \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") "
Sep 16 18:43:25.384490 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.384369    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-oauth-serving-cert\") pod \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\" (UID: \"b3756240-cc4f-4aa5-9e82-72e10cbf2a23\") "
Sep 16 18:43:25.384538 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.384493    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-config" (OuterVolumeSpecName: "console-config") pod "b3756240-cc4f-4aa5-9e82-72e10cbf2a23" (UID: "b3756240-cc4f-4aa5-9e82-72e10cbf2a23"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:25.384538 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.384519    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-service-ca" (OuterVolumeSpecName: "service-ca") pod "b3756240-cc4f-4aa5-9e82-72e10cbf2a23" (UID: "b3756240-cc4f-4aa5-9e82-72e10cbf2a23"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:25.384665 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.384647    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-service-ca\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:25.384723 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.384671    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:25.384894 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.384875    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b3756240-cc4f-4aa5-9e82-72e10cbf2a23" (UID: "b3756240-cc4f-4aa5-9e82-72e10cbf2a23"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:25.386614 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.386587    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b3756240-cc4f-4aa5-9e82-72e10cbf2a23" (UID: "b3756240-cc4f-4aa5-9e82-72e10cbf2a23"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:25.386883 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.386865    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b3756240-cc4f-4aa5-9e82-72e10cbf2a23" (UID: "b3756240-cc4f-4aa5-9e82-72e10cbf2a23"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:25.386980 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.386966    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-kube-api-access-hctkg" (OuterVolumeSpecName: "kube-api-access-hctkg") pod "b3756240-cc4f-4aa5-9e82-72e10cbf2a23" (UID: "b3756240-cc4f-4aa5-9e82-72e10cbf2a23"). InnerVolumeSpecName "kube-api-access-hctkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:25.485708 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.485687    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-oauth-serving-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:25.485708 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.485706    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hctkg\" (UniqueName: \"kubernetes.io/projected/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-kube-api-access-hctkg\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:25.485857 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.485716    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-serving-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:25.485857 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.485724    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b3756240-cc4f-4aa5-9e82-72e10cbf2a23-console-oauth-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:25.495629 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.495607    2583 generic.go:356] "Generic (PLEG): container finished" podID="b3756240-cc4f-4aa5-9e82-72e10cbf2a23" containerID="e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6" exitCode=0
Sep 16 18:43:25.495706 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.495645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4fcf8d74-hslq7" event={"ID":"b3756240-cc4f-4aa5-9e82-72e10cbf2a23","Type":"ContainerDied","Data":"e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6"}
Sep 16 18:43:25.495706 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.495651    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4fcf8d74-hslq7"
Sep 16 18:43:25.495706 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.495679    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4fcf8d74-hslq7" event={"ID":"b3756240-cc4f-4aa5-9e82-72e10cbf2a23","Type":"ContainerDied","Data":"526d36adf471c0c25ee4e10072c01ed1596fc0d105c1d3a05eec8e83118b2689"}
Sep 16 18:43:25.495706 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.495700    2583 scope.go:117] "RemoveContainer" containerID="e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6"
Sep 16 18:43:25.503930 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.503910    2583 scope.go:117] "RemoveContainer" containerID="e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6"
Sep 16 18:43:25.504194 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:43:25.504178    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6\": container with ID starting with e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6 not found: ID does not exist" containerID="e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6"
Sep 16 18:43:25.504242 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.504202    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6"} err="failed to get container status \"e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6\": rpc error: code = NotFound desc = could not find container \"e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6\": container with ID starting with e0f097f7c424039577b9636aebac204fcdc51b39681db5fdcb45012b060770a6 not found: ID does not exist"
Sep 16 18:43:25.517257 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.517234    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f4fcf8d74-hslq7"]
Sep 16 18:43:25.525770 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.525021    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f4fcf8d74-hslq7"]
Sep 16 18:43:25.658051 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:25.657996    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3756240-cc4f-4aa5-9e82-72e10cbf2a23" path="/var/lib/kubelet/pods/b3756240-cc4f-4aa5-9e82-72e10cbf2a23/volumes"
Sep 16 18:43:34.529707 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:34.529672    2583 generic.go:356] "Generic (PLEG): container finished" podID="7761e9ff-e826-4316-9a1a-876df7e23311" containerID="b07796a872379682efaa1df243947bb571f677d85a94c474a860e8ca0e7dfb93" exitCode=0
Sep 16 18:43:34.530087 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:34.529763    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jcr5s" event={"ID":"7761e9ff-e826-4316-9a1a-876df7e23311","Type":"ContainerDied","Data":"b07796a872379682efaa1df243947bb571f677d85a94c474a860e8ca0e7dfb93"}
Sep 16 18:43:34.530133 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:34.530124    2583 scope.go:117] "RemoveContainer" containerID="b07796a872379682efaa1df243947bb571f677d85a94c474a860e8ca0e7dfb93"
Sep 16 18:43:35.536438 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:35.536408    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jcr5s" event={"ID":"7761e9ff-e826-4316-9a1a-876df7e23311","Type":"ContainerStarted","Data":"c7e27d597879a0c77a61aa3caf3a537fcd1fb56fccf9470d1dc1d967341cc69a"}
Sep 16 18:43:35.561219 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:35.561192    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5nt49_f9947aa6-6269-453c-9a91-e95f47fffac3/serve-healthcheck-canary/0.log"
Sep 16 18:43:38.812919 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:38.812886    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:43:38.816568 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:38.816541    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-64bf7496bf-767ls"
Sep 16 18:43:39.530215 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.530177    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fcf944dc5-tlnjb" podUID="2f183772-94f0-4886-b14d-84981be01c8c" containerName="console" containerID="cri-o://39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7" gracePeriod=15
Sep 16 18:43:39.825900 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.825869    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:39.901443 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.901412    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8vtd\" (UniqueName: \"kubernetes.io/projected/2f183772-94f0-4886-b14d-84981be01c8c-kube-api-access-g8vtd\") pod \"2f183772-94f0-4886-b14d-84981be01c8c\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") "
Sep 16 18:43:39.901580 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.901464    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-trusted-ca-bundle\") pod \"2f183772-94f0-4886-b14d-84981be01c8c\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") "
Sep 16 18:43:39.901580 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.901498    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-service-ca\") pod \"2f183772-94f0-4886-b14d-84981be01c8c\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") "
Sep 16 18:43:39.901580 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.901570    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-console-config\") pod \"2f183772-94f0-4886-b14d-84981be01c8c\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") "
Sep 16 18:43:39.901778 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.901605    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-serving-cert\") pod \"2f183772-94f0-4886-b14d-84981be01c8c\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") "
Sep 16 18:43:39.901778 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.901630    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-oauth-config\") pod \"2f183772-94f0-4886-b14d-84981be01c8c\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") "
Sep 16 18:43:39.901778 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.901662    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-oauth-serving-cert\") pod \"2f183772-94f0-4886-b14d-84981be01c8c\" (UID: \"2f183772-94f0-4886-b14d-84981be01c8c\") "
Sep 16 18:43:39.901954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.901926    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2f183772-94f0-4886-b14d-84981be01c8c" (UID: "2f183772-94f0-4886-b14d-84981be01c8c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:39.902015 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.901966    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-console-config" (OuterVolumeSpecName: "console-config") pod "2f183772-94f0-4886-b14d-84981be01c8c" (UID: "2f183772-94f0-4886-b14d-84981be01c8c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:39.902015 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.901981    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-service-ca" (OuterVolumeSpecName: "service-ca") pod "2f183772-94f0-4886-b14d-84981be01c8c" (UID: "2f183772-94f0-4886-b14d-84981be01c8c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:39.902358 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.902240    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2f183772-94f0-4886-b14d-84981be01c8c" (UID: "2f183772-94f0-4886-b14d-84981be01c8c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:43:39.903717 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.903692    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f183772-94f0-4886-b14d-84981be01c8c-kube-api-access-g8vtd" (OuterVolumeSpecName: "kube-api-access-g8vtd") pod "2f183772-94f0-4886-b14d-84981be01c8c" (UID: "2f183772-94f0-4886-b14d-84981be01c8c"). InnerVolumeSpecName "kube-api-access-g8vtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:39.903836 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.903818    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2f183772-94f0-4886-b14d-84981be01c8c" (UID: "2f183772-94f0-4886-b14d-84981be01c8c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:39.903939 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:39.903920    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2f183772-94f0-4886-b14d-84981be01c8c" (UID: "2f183772-94f0-4886-b14d-84981be01c8c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:40.002929 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.002906    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8vtd\" (UniqueName: \"kubernetes.io/projected/2f183772-94f0-4886-b14d-84981be01c8c-kube-api-access-g8vtd\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:40.002929 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.002926    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-trusted-ca-bundle\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:40.003049 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.002935    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-service-ca\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:40.003049 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.002944    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-console-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:40.003049 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.002951    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-serving-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:40.003049 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.002959    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f183772-94f0-4886-b14d-84981be01c8c-console-oauth-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:40.003049 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.002967    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f183772-94f0-4886-b14d-84981be01c8c-oauth-serving-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:43:40.557015 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.556980    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f183772-94f0-4886-b14d-84981be01c8c" containerID="39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7" exitCode=0
Sep 16 18:43:40.557187 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.557028    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fcf944dc5-tlnjb"
Sep 16 18:43:40.557187 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.557063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcf944dc5-tlnjb" event={"ID":"2f183772-94f0-4886-b14d-84981be01c8c","Type":"ContainerDied","Data":"39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7"}
Sep 16 18:43:40.557187 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.557112    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fcf944dc5-tlnjb" event={"ID":"2f183772-94f0-4886-b14d-84981be01c8c","Type":"ContainerDied","Data":"1188f1e4cbde0f6383c0fcb121d23d3263e7a5ba6156ccc63f8235570af0bfa7"}
Sep 16 18:43:40.557187 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.557132    2583 scope.go:117] "RemoveContainer" containerID="39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7"
Sep 16 18:43:40.566255 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.566242    2583 scope.go:117] "RemoveContainer" containerID="39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7"
Sep 16 18:43:40.566525 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:43:40.566477    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7\": container with ID starting with 39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7 not found: ID does not exist" containerID="39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7"
Sep 16 18:43:40.566577 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.566533    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7"} err="failed to get container status \"39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7\": rpc error: code = NotFound desc = could not find container \"39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7\": container with ID starting with 39986c7d3f09151d791d9b6030a4f467c73a239907b835cc842a36c4f4232da7 not found: ID does not exist"
Sep 16 18:43:40.578842 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.578816    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fcf944dc5-tlnjb"]
Sep 16 18:43:40.583671 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:40.583654    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fcf944dc5-tlnjb"]
Sep 16 18:43:41.658207 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:43:41.658174    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f183772-94f0-4886-b14d-84981be01c8c" path="/var/lib/kubelet/pods/2f183772-94f0-4886-b14d-84981be01c8c/volumes"
Sep 16 18:44:00.843500 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:00.843461    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:00.864511 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:00.864481    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:01.663186 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:01.663157    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:18.886258 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:18.886224    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:18.887155 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:18.887100    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="prometheus" containerID="cri-o://f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76" gracePeriod=600
Sep 16 18:44:18.887492 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:18.887468    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy" containerID="cri-o://2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff" gracePeriod=600
Sep 16 18:44:18.887588 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:18.887488    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy-web" containerID="cri-o://8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d" gracePeriod=600
Sep 16 18:44:18.887588 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:18.887550    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="config-reloader" containerID="cri-o://0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f" gracePeriod=600
Sep 16 18:44:18.887685 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:18.887587    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d" gracePeriod=600
Sep 16 18:44:18.887685 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:18.887637    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="thanos-sidecar" containerID="cri-o://bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a" gracePeriod=600
Sep 16 18:44:19.149384 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.149358    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.312657 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312625    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.312657 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312659    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-serving-certs-ca-bundle\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.312912 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-tls\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.312912 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312695    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.312912 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312726    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-1\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.312912 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312788    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-db\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.312912 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312826    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.312912 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312876    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btqkh\" (UniqueName: \"kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-kube-api-access-btqkh\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.312912 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312909    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-kube-rbac-proxy\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312940    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-metrics-client-ca\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312970    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-grpc-tls\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.312995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-tls-assets\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313024    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-2\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313055    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config-out\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313104    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-web-config\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313127    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-trusted-ca-bundle\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313158    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:19.313269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313182    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-0\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313244    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-metrics-client-certs\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313779 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313270    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:19.313779 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313317    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-thanos-prometheus-http-client-file\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313779 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313524    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-kubelet-serving-ca-bundle\") pod \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\" (UID: \"9b0e5be6-ca89-4cc3-b09d-47b6f745a936\") "
Sep 16 18:44:19.313935 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313839    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.313935 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313860    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.313935 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.313886    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:19.314880 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.314378    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:19.314880 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.314776    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:19.315076 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.314936    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:19.315598 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.315335    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:19.315789 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.315734    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:19.316046 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.316017    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:19.317474 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.317451    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:19.317590 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.317445    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config" (OuterVolumeSpecName: "config") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:19.317590 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.317478    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-kube-api-access-btqkh" (OuterVolumeSpecName: "kube-api-access-btqkh") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "kube-api-access-btqkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:19.317857 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.317830    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:19.317928 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.317882    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:19.317928 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.317904    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:19.318248 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.318216    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:19.318355 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.318283    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:19.318355 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.318350    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config-out" (OuterVolumeSpecName: "config-out") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:44:19.319073 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.319054    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:19.328242 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.328223    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-web-config" (OuterVolumeSpecName: "web-config") pod "9b0e5be6-ca89-4cc3-b09d-47b6f745a936" (UID: "9b0e5be6-ca89-4cc3-b09d-47b6f745a936"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:19.414788 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414720    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414788 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414758    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414788 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414769    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-db\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414788 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414778    2583 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414788 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414787    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btqkh\" (UniqueName: \"kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-kube-api-access-btqkh\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414795    2583 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-kube-rbac-proxy\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414803    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-metrics-client-ca\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414812    2583 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-grpc-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414820    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-tls-assets\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414829    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414836    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-config-out\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414845    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-web-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414852    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414860    2583 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414868    2583 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-metrics-client-certs\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414876    2583 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414886    2583 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.414975 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.414894    2583 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9b0e5be6-ca89-4cc3-b09d-47b6f745a936-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:19.718094 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718065    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerID="8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d" exitCode=0
Sep 16 18:44:19.718094 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718087    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerID="2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff" exitCode=0
Sep 16 18:44:19.718094 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718095    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerID="8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d" exitCode=0
Sep 16 18:44:19.718306 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718101    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerID="bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a" exitCode=0
Sep 16 18:44:19.718306 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718106    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerID="0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f" exitCode=0
Sep 16 18:44:19.718306 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718112    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerID="f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76" exitCode=0
Sep 16 18:44:19.718306 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718148    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.718306 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718147    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerDied","Data":"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"}
Sep 16 18:44:19.718306 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerDied","Data":"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"}
Sep 16 18:44:19.718306 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerDied","Data":"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"}
Sep 16 18:44:19.718306 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718293    2583 scope.go:117] "RemoveContainer" containerID="8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"
Sep 16 18:44:19.718652 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerDied","Data":"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"}
Sep 16 18:44:19.718652 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerDied","Data":"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"}
Sep 16 18:44:19.718652 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718401    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerDied","Data":"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"}
Sep 16 18:44:19.718652 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.718442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9b0e5be6-ca89-4cc3-b09d-47b6f745a936","Type":"ContainerDied","Data":"54aa2bf191b147829e509231bc2da12ae3fec91ffe56d07bd8649ae4c011da6a"}
Sep 16 18:44:19.726508 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.726491    2583 scope.go:117] "RemoveContainer" containerID="2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"
Sep 16 18:44:19.733721 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.733706    2583 scope.go:117] "RemoveContainer" containerID="8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"
Sep 16 18:44:19.740934 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.740908    2583 scope.go:117] "RemoveContainer" containerID="bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"
Sep 16 18:44:19.741455 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.741431    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:19.745706 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.745688    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:19.748566 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.748540    2583 scope.go:117] "RemoveContainer" containerID="0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"
Sep 16 18:44:19.755167 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.755151    2583 scope.go:117] "RemoveContainer" containerID="f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"
Sep 16 18:44:19.762157 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.762144    2583 scope.go:117] "RemoveContainer" containerID="5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"
Sep 16 18:44:19.768405 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.768390    2583 scope.go:117] "RemoveContainer" containerID="8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"
Sep 16 18:44:19.768642 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:44:19.768626    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": container with ID starting with 8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d not found: ID does not exist" containerID="8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"
Sep 16 18:44:19.768687 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.768649    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"} err="failed to get container status \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": rpc error: code = NotFound desc = could not find container \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": container with ID starting with 8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d not found: ID does not exist"
Sep 16 18:44:19.768687 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.768665    2583 scope.go:117] "RemoveContainer" containerID="2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"
Sep 16 18:44:19.769029 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:44:19.769009    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": container with ID starting with 2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff not found: ID does not exist" containerID="2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"
Sep 16 18:44:19.769119 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.769036    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"} err="failed to get container status \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": rpc error: code = NotFound desc = could not find container \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": container with ID starting with 2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff not found: ID does not exist"
Sep 16 18:44:19.769119 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.769053    2583 scope.go:117] "RemoveContainer" containerID="8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"
Sep 16 18:44:19.769318 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:44:19.769298    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": container with ID starting with 8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d not found: ID does not exist" containerID="8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"
Sep 16 18:44:19.769423 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.769321    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"} err="failed to get container status \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": rpc error: code = NotFound desc = could not find container \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": container with ID starting with 8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d not found: ID does not exist"
Sep 16 18:44:19.769423 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.769337    2583 scope.go:117] "RemoveContainer" containerID="bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"
Sep 16 18:44:19.769644 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:44:19.769621    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": container with ID starting with bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a not found: ID does not exist" containerID="bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"
Sep 16 18:44:19.769718 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.769649    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"} err="failed to get container status \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": rpc error: code = NotFound desc = could not find container \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": container with ID starting with bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a not found: ID does not exist"
Sep 16 18:44:19.769718 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.769668    2583 scope.go:117] "RemoveContainer" containerID="0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"
Sep 16 18:44:19.769958 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:44:19.769935    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": container with ID starting with 0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f not found: ID does not exist" containerID="0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"
Sep 16 18:44:19.769999 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.769965    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"} err="failed to get container status \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": rpc error: code = NotFound desc = could not find container \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": container with ID starting with 0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f not found: ID does not exist"
Sep 16 18:44:19.769999 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.769986    2583 scope.go:117] "RemoveContainer" containerID="f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"
Sep 16 18:44:19.770229 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:44:19.770214    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": container with ID starting with f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76 not found: ID does not exist" containerID="f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"
Sep 16 18:44:19.770272 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.770231    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"} err="failed to get container status \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": rpc error: code = NotFound desc = could not find container \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": container with ID starting with f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76 not found: ID does not exist"
Sep 16 18:44:19.770272 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.770243    2583 scope.go:117] "RemoveContainer" containerID="5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"
Sep 16 18:44:19.770499 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:44:19.770482    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": container with ID starting with 5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab not found: ID does not exist" containerID="5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"
Sep 16 18:44:19.770562 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.770507    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"} err="failed to get container status \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": rpc error: code = NotFound desc = could not find container \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": container with ID starting with 5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab not found: ID does not exist"
Sep 16 18:44:19.770562 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.770530    2583 scope.go:117] "RemoveContainer" containerID="8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"
Sep 16 18:44:19.770781 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.770766    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"} err="failed to get container status \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": rpc error: code = NotFound desc = could not find container \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": container with ID starting with 8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d not found: ID does not exist"
Sep 16 18:44:19.770781 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.770781    2583 scope.go:117] "RemoveContainer" containerID="2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"
Sep 16 18:44:19.771004 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.770985    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"} err="failed to get container status \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": rpc error: code = NotFound desc = could not find container \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": container with ID starting with 2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff not found: ID does not exist"
Sep 16 18:44:19.771045 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.771006    2583 scope.go:117] "RemoveContainer" containerID="8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"
Sep 16 18:44:19.771218 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.771204    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"} err="failed to get container status \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": rpc error: code = NotFound desc = could not find container \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": container with ID starting with 8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d not found: ID does not exist"
Sep 16 18:44:19.771271 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.771218    2583 scope.go:117] "RemoveContainer" containerID="bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"
Sep 16 18:44:19.771460 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.771436    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"} err="failed to get container status \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": rpc error: code = NotFound desc = could not find container \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": container with ID starting with bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a not found: ID does not exist"
Sep 16 18:44:19.771516 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.771463    2583 scope.go:117] "RemoveContainer" containerID="0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"
Sep 16 18:44:19.771691 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.771668    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"} err="failed to get container status \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": rpc error: code = NotFound desc = could not find container \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": container with ID starting with 0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f not found: ID does not exist"
Sep 16 18:44:19.771783 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.771694    2583 scope.go:117] "RemoveContainer" containerID="f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"
Sep 16 18:44:19.771960 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.771942    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"} err="failed to get container status \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": rpc error: code = NotFound desc = could not find container \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": container with ID starting with f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76 not found: ID does not exist"
Sep 16 18:44:19.772012 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.771960    2583 scope.go:117] "RemoveContainer" containerID="5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"
Sep 16 18:44:19.772200 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.772183    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"} err="failed to get container status \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": rpc error: code = NotFound desc = could not find container \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": container with ID starting with 5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab not found: ID does not exist"
Sep 16 18:44:19.772269 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.772203    2583 scope.go:117] "RemoveContainer" containerID="8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"
Sep 16 18:44:19.772428 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.772409    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"} err="failed to get container status \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": rpc error: code = NotFound desc = could not find container \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": container with ID starting with 8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d not found: ID does not exist"
Sep 16 18:44:19.772466 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.772429    2583 scope.go:117] "RemoveContainer" containerID="2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"
Sep 16 18:44:19.772654 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.772636    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"} err="failed to get container status \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": rpc error: code = NotFound desc = could not find container \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": container with ID starting with 2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff not found: ID does not exist"
Sep 16 18:44:19.772697 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.772654    2583 scope.go:117] "RemoveContainer" containerID="8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"
Sep 16 18:44:19.772871 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.772855    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"} err="failed to get container status \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": rpc error: code = NotFound desc = could not find container \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": container with ID starting with 8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d not found: ID does not exist"
Sep 16 18:44:19.772914 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.772870    2583 scope.go:117] "RemoveContainer" containerID="bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"
Sep 16 18:44:19.773065 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.773049    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"} err="failed to get container status \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": rpc error: code = NotFound desc = could not find container \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": container with ID starting with bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a not found: ID does not exist"
Sep 16 18:44:19.773112 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.773065    2583 scope.go:117] "RemoveContainer" containerID="0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"
Sep 16 18:44:19.773287 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.773272    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"} err="failed to get container status \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": rpc error: code = NotFound desc = could not find container \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": container with ID starting with 0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f not found: ID does not exist"
Sep 16 18:44:19.773336 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.773287    2583 scope.go:117] "RemoveContainer" containerID="f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"
Sep 16 18:44:19.773485 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.773468    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"} err="failed to get container status \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": rpc error: code = NotFound desc = could not find container \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": container with ID starting with f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76 not found: ID does not exist"
Sep 16 18:44:19.773524 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.773486    2583 scope.go:117] "RemoveContainer" containerID="5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"
Sep 16 18:44:19.773730 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.773711    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"} err="failed to get container status \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": rpc error: code = NotFound desc = could not find container \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": container with ID starting with 5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab not found: ID does not exist"
Sep 16 18:44:19.773730 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.773729    2583 scope.go:117] "RemoveContainer" containerID="8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"
Sep 16 18:44:19.773942 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.773926    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"} err="failed to get container status \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": rpc error: code = NotFound desc = could not find container \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": container with ID starting with 8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d not found: ID does not exist"
Sep 16 18:44:19.773942 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.773940    2583 scope.go:117] "RemoveContainer" containerID="2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"
Sep 16 18:44:19.774130 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.774115    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"} err="failed to get container status \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": rpc error: code = NotFound desc = could not find container \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": container with ID starting with 2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff not found: ID does not exist"
Sep 16 18:44:19.774130 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.774129    2583 scope.go:117] "RemoveContainer" containerID="8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"
Sep 16 18:44:19.774312 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.774297    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"} err="failed to get container status \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": rpc error: code = NotFound desc = could not find container \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": container with ID starting with 8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d not found: ID does not exist"
Sep 16 18:44:19.774312 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.774311    2583 scope.go:117] "RemoveContainer" containerID="bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"
Sep 16 18:44:19.774534 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.774519    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"} err="failed to get container status \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": rpc error: code = NotFound desc = could not find container \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": container with ID starting with bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a not found: ID does not exist"
Sep 16 18:44:19.774591 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.774535    2583 scope.go:117] "RemoveContainer" containerID="0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"
Sep 16 18:44:19.774790 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.774772    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"} err="failed to get container status \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": rpc error: code = NotFound desc = could not find container \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": container with ID starting with 0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f not found: ID does not exist"
Sep 16 18:44:19.774840 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.774791    2583 scope.go:117] "RemoveContainer" containerID="f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"
Sep 16 18:44:19.775015 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.774998    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"} err="failed to get container status \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": rpc error: code = NotFound desc = could not find container \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": container with ID starting with f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76 not found: ID does not exist"
Sep 16 18:44:19.775057 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.775016    2583 scope.go:117] "RemoveContainer" containerID="5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"
Sep 16 18:44:19.775247 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.775232    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"} err="failed to get container status \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": rpc error: code = NotFound desc = could not find container \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": container with ID starting with 5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab not found: ID does not exist"
Sep 16 18:44:19.775287 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.775247    2583 scope.go:117] "RemoveContainer" containerID="8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"
Sep 16 18:44:19.775461 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.775444    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"} err="failed to get container status \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": rpc error: code = NotFound desc = could not find container \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": container with ID starting with 8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d not found: ID does not exist"
Sep 16 18:44:19.775507 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.775462    2583 scope.go:117] "RemoveContainer" containerID="2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"
Sep 16 18:44:19.775837 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.775815    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"} err="failed to get container status \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": rpc error: code = NotFound desc = could not find container \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": container with ID starting with 2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff not found: ID does not exist"
Sep 16 18:44:19.775837 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.775836    2583 scope.go:117] "RemoveContainer" containerID="8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"
Sep 16 18:44:19.776170 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.776133    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"} err="failed to get container status \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": rpc error: code = NotFound desc = could not find container \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": container with ID starting with 8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d not found: ID does not exist"
Sep 16 18:44:19.776283 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.776171    2583 scope.go:117] "RemoveContainer" containerID="bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"
Sep 16 18:44:19.776527 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.776503    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"} err="failed to get container status \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": rpc error: code = NotFound desc = could not find container \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": container with ID starting with bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a not found: ID does not exist"
Sep 16 18:44:19.776615 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.776528    2583 scope.go:117] "RemoveContainer" containerID="0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"
Sep 16 18:44:19.776868 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.776841    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"} err="failed to get container status \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": rpc error: code = NotFound desc = could not find container \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": container with ID starting with 0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f not found: ID does not exist"
Sep 16 18:44:19.776868 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.776867    2583 scope.go:117] "RemoveContainer" containerID="f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"
Sep 16 18:44:19.777179 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.777154    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"} err="failed to get container status \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": rpc error: code = NotFound desc = could not find container \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": container with ID starting with f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76 not found: ID does not exist"
Sep 16 18:44:19.777179 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.777180    2583 scope.go:117] "RemoveContainer" containerID="5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"
Sep 16 18:44:19.777466 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.777447    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"} err="failed to get container status \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": rpc error: code = NotFound desc = could not find container \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": container with ID starting with 5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab not found: ID does not exist"
Sep 16 18:44:19.777466 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.777466    2583 scope.go:117] "RemoveContainer" containerID="8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"
Sep 16 18:44:19.777694 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.777676    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:19.777779 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.777675    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d"} err="failed to get container status \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": rpc error: code = NotFound desc = could not find container \"8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d\": container with ID starting with 8b3775a69dad618f32b8268997179e50554b224c38ddb03ab8a4a0a0cb08898d not found: ID does not exist"
Sep 16 18:44:19.777779 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.777775    2583 scope.go:117] "RemoveContainer" containerID="2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"
Sep 16 18:44:19.778002 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.777987    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="thanos-sidecar"
Sep 16 18:44:19.778070 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.777995    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff"} err="failed to get container status \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": rpc error: code = NotFound desc = could not find container \"2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff\": container with ID starting with 2c65b78dcd608f13efd52f118000028b625432949947c2f8078b5417154548ff not found: ID does not exist"
Sep 16 18:44:19.778070 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778015    2583 scope.go:117] "RemoveContainer" containerID="8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"
Sep 16 18:44:19.778070 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778003    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="thanos-sidecar"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778081    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy-thanos"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778092    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy-thanos"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778108    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="config-reloader"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778117    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="config-reloader"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778125    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778134    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778145    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="init-config-reloader"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778154    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="init-config-reloader"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778174    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="prometheus"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778183    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="prometheus"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778198    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f183772-94f0-4886-b14d-84981be01c8c" containerName="console"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778206    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f183772-94f0-4886-b14d-84981be01c8c" containerName="console"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778217    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy-web"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778226    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy-web"
Sep 16 18:44:19.778228 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778227    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d"} err="failed to get container status \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": rpc error: code = NotFound desc = could not find container \"8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d\": container with ID starting with 8cc8b6300b6fa1521e638594fd40a7f6d0cd213de04773683758982ae55c108d not found: ID does not exist"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778237    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3756240-cc4f-4aa5-9e82-72e10cbf2a23" containerName="console"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778246    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3756240-cc4f-4aa5-9e82-72e10cbf2a23" containerName="console"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778240    2583 scope.go:117] "RemoveContainer" containerID="bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778334    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="config-reloader"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778348    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="thanos-sidecar"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778358    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f183772-94f0-4886-b14d-84981be01c8c" containerName="console"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778366    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy-web"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778376    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778385    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="kube-rbac-proxy-thanos"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778393    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3756240-cc4f-4aa5-9e82-72e10cbf2a23" containerName="console"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778402    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" containerName="prometheus"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778493    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a"} err="failed to get container status \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": rpc error: code = NotFound desc = could not find container \"bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a\": container with ID starting with bfd044d415a9b5c1018aeb35e07bdae211e69ca6e999f16dd819b9024283424a not found: ID does not exist"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778507    2583 scope.go:117] "RemoveContainer" containerID="0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778762    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f"} err="failed to get container status \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": rpc error: code = NotFound desc = could not find container \"0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f\": container with ID starting with 0166a79d6fc2f83ae2433a92123c75c3d7feebb08410aef16d1ae190f52f882f not found: ID does not exist"
Sep 16 18:44:19.778954 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778778    2583 scope.go:117] "RemoveContainer" containerID="f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"
Sep 16 18:44:19.779660 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778965    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76"} err="failed to get container status \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": rpc error: code = NotFound desc = could not find container \"f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76\": container with ID starting with f9474a51e6eec1243a0f0e170a542532098a4e67bca90c4e39ae06d679b5ab76 not found: ID does not exist"
Sep 16 18:44:19.779660 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.778977    2583 scope.go:117] "RemoveContainer" containerID="5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"
Sep 16 18:44:19.779660 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.779210    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab"} err="failed to get container status \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": rpc error: code = NotFound desc = could not find container \"5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab\": container with ID starting with 5cf2bdc2183325ccb730f8a14646a7f9a6d7cc3e30d2eaefaa3b6077d72763ab not found: ID does not exist"
Sep 16 18:44:19.784060 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.784042    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.786255 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.786236    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:44:19.786330 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.786279    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7akr39utptdeq\""
Sep 16 18:44:19.786815 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.786802    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:44:19.786902 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.786887    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:44:19.786967 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.786896    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:44:19.786967 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.786908    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:44:19.786967 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.786935    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:44:19.787260 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.787095    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:44:19.787260 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.787095    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rrj8v\""
Sep 16 18:44:19.787360 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.787273    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:44:19.787360 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.787320    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:44:19.787458 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.787405    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:44:19.787530 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.787516    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:44:19.787587 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.787536    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:44:19.789900 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.789854    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:44:19.796119 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.794612    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:44:19.798998 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.798974    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:19.919368 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919368 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919338    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-config\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919612    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919628    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919653    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-config-out\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.919714 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919711    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.920114 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-web-config\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.920114 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919764    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.920114 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.920114 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.920114 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:19.920114 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:19.919886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57q22\" (UniqueName: \"kubernetes.io/projected/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-kube-api-access-57q22\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020349 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020432 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-config\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020432 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020432 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020432 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020432 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020452    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-config-out\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-web-config\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020611    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.020715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57q22\" (UniqueName: \"kubernetes.io/projected/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-kube-api-access-57q22\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.021428 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.021428 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.021428 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020836    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.021428 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.020860    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.021428 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.021158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.021686 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.021605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.021686 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.021631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.021820 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.021773    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.023991 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.023880    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.023991 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.023889    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.023991 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.023889    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-config\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.024767 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.024472    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.024767 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.024577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.024767 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.024556    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.024767 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.024682    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.025043 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.024841    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.025478 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.025454    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.025885 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.025864    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.026808 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.026781    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.026921 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.026900    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-config-out\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.026960 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.026941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-web-config\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.027060 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.027046    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.031264 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.031244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57q22\" (UniqueName: \"kubernetes.io/projected/f82e57f8-4c77-4a9a-bea8-85f1b1394d3b-kube-api-access-57q22\") pod \"prometheus-k8s-0\" (UID: \"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.098542 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.098519    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:20.379421 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.379399    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:44:20.381188 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:44:20.381156    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82e57f8_4c77_4a9a_bea8_85f1b1394d3b.slice/crio-98c3278a6ed521f07f6e3e67c4ada7acf4d2f4232ba63aa10fd4fbf510aa8ba7 WatchSource:0}: Error finding container 98c3278a6ed521f07f6e3e67c4ada7acf4d2f4232ba63aa10fd4fbf510aa8ba7: Status 404 returned error can't find the container with id 98c3278a6ed521f07f6e3e67c4ada7acf4d2f4232ba63aa10fd4fbf510aa8ba7
Sep 16 18:44:20.725101 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.725067    2583 generic.go:356] "Generic (PLEG): container finished" podID="f82e57f8-4c77-4a9a-bea8-85f1b1394d3b" containerID="68fba1c335804417f59134db95cc6b0e52967240f377e1ed99cf60fc88f76c80" exitCode=0
Sep 16 18:44:20.725216 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.725110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b","Type":"ContainerDied","Data":"68fba1c335804417f59134db95cc6b0e52967240f377e1ed99cf60fc88f76c80"}
Sep 16 18:44:20.725216 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:20.725183    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b","Type":"ContainerStarted","Data":"98c3278a6ed521f07f6e3e67c4ada7acf4d2f4232ba63aa10fd4fbf510aa8ba7"}
Sep 16 18:44:21.012546 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:21.012518    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-674957ddc8-767mt"]
Sep 16 18:44:21.659327 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:21.659293    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0e5be6-ca89-4cc3-b09d-47b6f745a936" path="/var/lib/kubelet/pods/9b0e5be6-ca89-4cc3-b09d-47b6f745a936/volumes"
Sep 16 18:44:21.733862 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:21.733826    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b","Type":"ContainerStarted","Data":"7594585f0412e7ba631def908bb4892cb273340ae95c98b3d7a8ab13e5d17068"}
Sep 16 18:44:21.733998 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:21.733903    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b","Type":"ContainerStarted","Data":"c44b8959b44ac04551423652307cda68ed4efd27ce5337c26b78e812d5947bcf"}
Sep 16 18:44:21.733998 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:21.733943    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b","Type":"ContainerStarted","Data":"77c6d1a2445328ac05100ee6e166b79f2fe42ea2270e78c4514574fb64f9f588"}
Sep 16 18:44:21.733998 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:21.733982    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b","Type":"ContainerStarted","Data":"2ebc9ab8e3c4fe73c0430a56f7dff10d652ed6379935f093d2fe026eb2e576da"}
Sep 16 18:44:21.734115 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:21.734020    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b","Type":"ContainerStarted","Data":"41952fc7dd4cb546f702a45240b104d850d7166395027a142317090981d0bf34"}
Sep 16 18:44:21.734115 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:21.734054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82e57f8-4c77-4a9a-bea8-85f1b1394d3b","Type":"ContainerStarted","Data":"9b951f41da6480377ef8789e507ea9695854b9822907fb68e878044cb34d98d2"}
Sep 16 18:44:21.762669 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:21.762621    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.743626562 podStartE2EDuration="2.762602668s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.976106ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:21.760948372 +0000 UTC m=+234.643771454" watchObservedRunningTime="2026-09-16 18:44:21.762602668 +0000 UTC m=+234.645425750"
Sep 16 18:44:25.098921 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:25.098880    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:44:46.036818 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.036772    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-674957ddc8-767mt" podUID="e85973ce-26d0-4619-b38d-7ba9cfc474f4" containerName="console" containerID="cri-o://4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6" gracePeriod=15
Sep 16 18:44:46.290023 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.289974    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:44:46.323077 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.323050    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-trusted-ca-bundle\") pod \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") "
Sep 16 18:44:46.323173 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.323082    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-oauth-serving-cert\") pod \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") "
Sep 16 18:44:46.323173 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.323132    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-oauth-config\") pod \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") "
Sep 16 18:44:46.323173 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.323154    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-serving-cert\") pod \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") "
Sep 16 18:44:46.323323 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.323186    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-config\") pod \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") "
Sep 16 18:44:46.323323 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.323231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-service-ca\") pod \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") "
Sep 16 18:44:46.323323 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.323270    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksbnb\" (UniqueName: \"kubernetes.io/projected/e85973ce-26d0-4619-b38d-7ba9cfc474f4-kube-api-access-ksbnb\") pod \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\" (UID: \"e85973ce-26d0-4619-b38d-7ba9cfc474f4\") "
Sep 16 18:44:46.323563 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.323442    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e85973ce-26d0-4619-b38d-7ba9cfc474f4" (UID: "e85973ce-26d0-4619-b38d-7ba9cfc474f4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:46.323563 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.323551    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e85973ce-26d0-4619-b38d-7ba9cfc474f4" (UID: "e85973ce-26d0-4619-b38d-7ba9cfc474f4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:46.323734 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.323620    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-config" (OuterVolumeSpecName: "console-config") pod "e85973ce-26d0-4619-b38d-7ba9cfc474f4" (UID: "e85973ce-26d0-4619-b38d-7ba9cfc474f4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:46.323826 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.323761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-service-ca" (OuterVolumeSpecName: "service-ca") pod "e85973ce-26d0-4619-b38d-7ba9cfc474f4" (UID: "e85973ce-26d0-4619-b38d-7ba9cfc474f4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:44:46.325715 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.325688    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85973ce-26d0-4619-b38d-7ba9cfc474f4-kube-api-access-ksbnb" (OuterVolumeSpecName: "kube-api-access-ksbnb") pod "e85973ce-26d0-4619-b38d-7ba9cfc474f4" (UID: "e85973ce-26d0-4619-b38d-7ba9cfc474f4"). InnerVolumeSpecName "kube-api-access-ksbnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:44:46.325994 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.325960    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e85973ce-26d0-4619-b38d-7ba9cfc474f4" (UID: "e85973ce-26d0-4619-b38d-7ba9cfc474f4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:46.325994 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.325972    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e85973ce-26d0-4619-b38d-7ba9cfc474f4" (UID: "e85973ce-26d0-4619-b38d-7ba9cfc474f4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:44:46.424506 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.424485    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksbnb\" (UniqueName: \"kubernetes.io/projected/e85973ce-26d0-4619-b38d-7ba9cfc474f4-kube-api-access-ksbnb\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:46.424506 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.424505    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-trusted-ca-bundle\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:46.424616 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.424515    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-oauth-serving-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:46.424616 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.424534    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-oauth-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:46.424616 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.424542    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-serving-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:46.424616 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.424555    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-console-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:46.424616 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.424567    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85973ce-26d0-4619-b38d-7ba9cfc474f4-service-ca\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:44:46.838978 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.838944    2583 generic.go:356] "Generic (PLEG): container finished" podID="e85973ce-26d0-4619-b38d-7ba9cfc474f4" containerID="4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6" exitCode=0
Sep 16 18:44:46.839122 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.838993    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-674957ddc8-767mt"
Sep 16 18:44:46.839122 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.839042    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674957ddc8-767mt" event={"ID":"e85973ce-26d0-4619-b38d-7ba9cfc474f4","Type":"ContainerDied","Data":"4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6"}
Sep 16 18:44:46.839122 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.839096    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-674957ddc8-767mt" event={"ID":"e85973ce-26d0-4619-b38d-7ba9cfc474f4","Type":"ContainerDied","Data":"cc3fda66dbe5bae0996287577b1565d0b17d7a65aabca86ae2e17afc9f7a0134"}
Sep 16 18:44:46.839122 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.839118    2583 scope.go:117] "RemoveContainer" containerID="4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6"
Sep 16 18:44:46.848203 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.848185    2583 scope.go:117] "RemoveContainer" containerID="4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6"
Sep 16 18:44:46.848503 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:44:46.848486    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6\": container with ID starting with 4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6 not found: ID does not exist" containerID="4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6"
Sep 16 18:44:46.848555 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.848511    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6"} err="failed to get container status \"4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6\": rpc error: code = NotFound desc = could not find container \"4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6\": container with ID starting with 4062fc82fd72c459bda4e899a80cfcfd8e132ae40231cb6491b683013fc815f6 not found: ID does not exist"
Sep 16 18:44:46.860918 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.860894    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-674957ddc8-767mt"]
Sep 16 18:44:46.865314 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:46.865294    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-674957ddc8-767mt"]
Sep 16 18:44:47.662609 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:44:47.662566    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e85973ce-26d0-4619-b38d-7ba9cfc474f4" path="/var/lib/kubelet/pods/e85973ce-26d0-4619-b38d-7ba9cfc474f4/volumes"
Sep 16 18:45:20.099177 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:45:20.099144    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:20.114497 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:45:20.114471    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:20.987885 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:45:20.987853    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:45:27.548366 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:45:27.548333    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:47:55.430007 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.429926    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74f6bf7ddb-l7lld"]
Sep 16 18:47:55.430491 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.430251    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e85973ce-26d0-4619-b38d-7ba9cfc474f4" containerName="console"
Sep 16 18:47:55.430491 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.430265    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85973ce-26d0-4619-b38d-7ba9cfc474f4" containerName="console"
Sep 16 18:47:55.430491 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.430318    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e85973ce-26d0-4619-b38d-7ba9cfc474f4" containerName="console"
Sep 16 18:47:55.433244 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.433224    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.436317 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.436291    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:47:55.436471 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.436291    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hxfv9\""
Sep 16 18:47:55.436471 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.436349    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:47:55.436471 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.436412    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:47:55.436471 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.436412    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:47:55.437182 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.437159    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:47:55.437330 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.437198    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:47:55.437330 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.437297    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:47:55.441724 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.441704    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:47:55.442284 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.442253    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f6bf7ddb-l7lld"]
Sep 16 18:47:55.519398 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.519364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-trusted-ca-bundle\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.519560 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.519420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxgt\" (UniqueName: \"kubernetes.io/projected/fe2097f6-50b1-4311-843c-108cbec24d7f-kube-api-access-zjxgt\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.519560 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.519447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-service-ca\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.519560 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.519487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe2097f6-50b1-4311-843c-108cbec24d7f-console-serving-cert\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.519560 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.519519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-oauth-serving-cert\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.519560 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.519538    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe2097f6-50b1-4311-843c-108cbec24d7f-console-oauth-config\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.519784 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.519619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-console-config\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.620732 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.620708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-console-config\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.620901 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.620757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-trusted-ca-bundle\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.620901 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.620792    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxgt\" (UniqueName: \"kubernetes.io/projected/fe2097f6-50b1-4311-843c-108cbec24d7f-kube-api-access-zjxgt\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.620901 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.620808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-service-ca\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.620901 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.620831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe2097f6-50b1-4311-843c-108cbec24d7f-console-serving-cert\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.620901 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.620858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-oauth-serving-cert\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.620901 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.620882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe2097f6-50b1-4311-843c-108cbec24d7f-console-oauth-config\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.621549 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.621525    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-service-ca\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.621670 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.621528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-console-config\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.621670 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.621657    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-oauth-serving-cert\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.621864 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.621848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe2097f6-50b1-4311-843c-108cbec24d7f-trusted-ca-bundle\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.623430 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.623401    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe2097f6-50b1-4311-843c-108cbec24d7f-console-serving-cert\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.623615 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.623595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe2097f6-50b1-4311-843c-108cbec24d7f-console-oauth-config\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.630171 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.630152    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxgt\" (UniqueName: \"kubernetes.io/projected/fe2097f6-50b1-4311-843c-108cbec24d7f-kube-api-access-zjxgt\") pod \"console-74f6bf7ddb-l7lld\" (UID: \"fe2097f6-50b1-4311-843c-108cbec24d7f\") " pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:55.743553 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:55.743470    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:47:56.017239 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:56.017109    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f6bf7ddb-l7lld"]
Sep 16 18:47:56.019962 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:47:56.019930    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2097f6_50b1_4311_843c_108cbec24d7f.slice/crio-66c84a8928f3cf69e6aa22ffd9e5eb4955be2cab106e9bf5a6b9311d1760c464 WatchSource:0}: Error finding container 66c84a8928f3cf69e6aa22ffd9e5eb4955be2cab106e9bf5a6b9311d1760c464: Status 404 returned error can't find the container with id 66c84a8928f3cf69e6aa22ffd9e5eb4955be2cab106e9bf5a6b9311d1760c464
Sep 16 18:47:56.571915 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:56.571878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f6bf7ddb-l7lld" event={"ID":"fe2097f6-50b1-4311-843c-108cbec24d7f","Type":"ContainerStarted","Data":"36a859557e984c64dd827442693587b526cdd8d3e46af5a77ecf842fab45aeb8"}
Sep 16 18:47:56.572288 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:56.571955    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f6bf7ddb-l7lld" event={"ID":"fe2097f6-50b1-4311-843c-108cbec24d7f","Type":"ContainerStarted","Data":"66c84a8928f3cf69e6aa22ffd9e5eb4955be2cab106e9bf5a6b9311d1760c464"}
Sep 16 18:47:56.591082 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:47:56.591035    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74f6bf7ddb-l7lld" podStartSLOduration=1.59101972 podStartE2EDuration="1.59101972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:56.5902558 +0000 UTC m=+449.473078881" watchObservedRunningTime="2026-09-16 18:47:56.59101972 +0000 UTC m=+449.473842800"
Sep 16 18:48:05.744499 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:48:05.744469    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:48:05.744863 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:48:05.744516    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:48:05.749034 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:48:05.749017    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:48:06.615083 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:48:06.615057    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74f6bf7ddb-l7lld"
Sep 16 18:49:10.131360 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.131332    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-s9fqh"]
Sep 16 18:49:10.134790 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.134772    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-s9fqh"
Sep 16 18:49:10.137254 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.137234    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4rtmh\""
Sep 16 18:49:10.137345 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.137261    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:49:10.137895 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.137875    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:49:10.137990 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.137907    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:49:10.137990 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.137907    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:49:10.142187 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.142164    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-s9fqh"]
Sep 16 18:49:10.180719 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.180692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sql2c\" (UniqueName: \"kubernetes.io/projected/3c1f70ee-4ca7-4550-b098-bcd450197a28-kube-api-access-sql2c\") pod \"s3-init-s9fqh\" (UID: \"3c1f70ee-4ca7-4550-b098-bcd450197a28\") " pod="kserve/s3-init-s9fqh"
Sep 16 18:49:10.180852 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.180726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c1f70ee-4ca7-4550-b098-bcd450197a28-model-cache\") pod \"s3-init-s9fqh\" (UID: \"3c1f70ee-4ca7-4550-b098-bcd450197a28\") " pod="kserve/s3-init-s9fqh"
Sep 16 18:49:10.281403 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.281383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sql2c\" (UniqueName: \"kubernetes.io/projected/3c1f70ee-4ca7-4550-b098-bcd450197a28-kube-api-access-sql2c\") pod \"s3-init-s9fqh\" (UID: \"3c1f70ee-4ca7-4550-b098-bcd450197a28\") " pod="kserve/s3-init-s9fqh"
Sep 16 18:49:10.281502 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.281409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c1f70ee-4ca7-4550-b098-bcd450197a28-model-cache\") pod \"s3-init-s9fqh\" (UID: \"3c1f70ee-4ca7-4550-b098-bcd450197a28\") " pod="kserve/s3-init-s9fqh"
Sep 16 18:49:10.281729 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.281712    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c1f70ee-4ca7-4550-b098-bcd450197a28-model-cache\") pod \"s3-init-s9fqh\" (UID: \"3c1f70ee-4ca7-4550-b098-bcd450197a28\") " pod="kserve/s3-init-s9fqh"
Sep 16 18:49:10.291701 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.291684    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sql2c\" (UniqueName: \"kubernetes.io/projected/3c1f70ee-4ca7-4550-b098-bcd450197a28-kube-api-access-sql2c\") pod \"s3-init-s9fqh\" (UID: \"3c1f70ee-4ca7-4550-b098-bcd450197a28\") " pod="kserve/s3-init-s9fqh"
Sep 16 18:49:10.445561 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.445526    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-s9fqh"
Sep 16 18:49:10.722730 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.722655    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-s9fqh"]
Sep 16 18:49:10.725759 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:49:10.725715    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c1f70ee_4ca7_4550_b098_bcd450197a28.slice/crio-43a64a9c67635eb5cfc76f986163d072f2feab3c65ef70adb5e3014ba8ff8892 WatchSource:0}: Error finding container 43a64a9c67635eb5cfc76f986163d072f2feab3c65ef70adb5e3014ba8ff8892: Status 404 returned error can't find the container with id 43a64a9c67635eb5cfc76f986163d072f2feab3c65ef70adb5e3014ba8ff8892
Sep 16 18:49:10.727848 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.727829    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:49:10.860637 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:10.860614    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s9fqh" event={"ID":"3c1f70ee-4ca7-4550-b098-bcd450197a28","Type":"ContainerStarted","Data":"43a64a9c67635eb5cfc76f986163d072f2feab3c65ef70adb5e3014ba8ff8892"}
Sep 16 18:49:14.882256 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:14.882219    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s9fqh" event={"ID":"3c1f70ee-4ca7-4550-b098-bcd450197a28","Type":"ContainerStarted","Data":"d4b537f4bf23c6105a297c131efb60f96a2a367c96f01641bd35c51f1ae9b2ed"}
Sep 16 18:49:15.886803 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:15.886776    2583 generic.go:356] "Generic (PLEG): container finished" podID="3c1f70ee-4ca7-4550-b098-bcd450197a28" containerID="d4b537f4bf23c6105a297c131efb60f96a2a367c96f01641bd35c51f1ae9b2ed" exitCode=0
Sep 16 18:49:15.887115 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:15.886865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s9fqh" event={"ID":"3c1f70ee-4ca7-4550-b098-bcd450197a28","Type":"ContainerDied","Data":"d4b537f4bf23c6105a297c131efb60f96a2a367c96f01641bd35c51f1ae9b2ed"}
Sep 16 18:49:20.915517 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:20.915482    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s9fqh" event={"ID":"3c1f70ee-4ca7-4550-b098-bcd450197a28","Type":"ContainerStarted","Data":"c35df9e829a5d434634f29fd3c9f82be2ba76a9b406f070b0dfe645e5bb5282f"}
Sep 16 18:49:20.933348 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:49:20.933304    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-s9fqh" podStartSLOduration=2.016044999 podStartE2EDuration="10.933285659s" totalImagesPullingTime="7.582182714s" totalInitContainerRuntime="1.335057946s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:20.932831164 +0000 UTC m=+533.815654245" watchObservedRunningTime="2026-09-16 18:49:20.933285659 +0000 UTC m=+533.816108759"
Sep 16 18:50:26.173435 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:26.173402    2583 generic.go:356] "Generic (PLEG): container finished" podID="3c1f70ee-4ca7-4550-b098-bcd450197a28" containerID="c35df9e829a5d434634f29fd3c9f82be2ba76a9b406f070b0dfe645e5bb5282f" exitCode=0
Sep 16 18:50:26.173844 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:26.173476    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s9fqh" event={"ID":"3c1f70ee-4ca7-4550-b098-bcd450197a28","Type":"ContainerDied","Data":"c35df9e829a5d434634f29fd3c9f82be2ba76a9b406f070b0dfe645e5bb5282f"}
Sep 16 18:50:27.297433 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:27.297408    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-s9fqh"
Sep 16 18:50:27.411803 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:27.411777    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sql2c\" (UniqueName: \"kubernetes.io/projected/3c1f70ee-4ca7-4550-b098-bcd450197a28-kube-api-access-sql2c\") pod \"3c1f70ee-4ca7-4550-b098-bcd450197a28\" (UID: \"3c1f70ee-4ca7-4550-b098-bcd450197a28\") "
Sep 16 18:50:27.411931 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:27.411854    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c1f70ee-4ca7-4550-b098-bcd450197a28-model-cache\") pod \"3c1f70ee-4ca7-4550-b098-bcd450197a28\" (UID: \"3c1f70ee-4ca7-4550-b098-bcd450197a28\") "
Sep 16 18:50:27.414113 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:27.414084    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1f70ee-4ca7-4550-b098-bcd450197a28-kube-api-access-sql2c" (OuterVolumeSpecName: "kube-api-access-sql2c") pod "3c1f70ee-4ca7-4550-b098-bcd450197a28" (UID: "3c1f70ee-4ca7-4550-b098-bcd450197a28"). InnerVolumeSpecName "kube-api-access-sql2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:27.430107 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:27.430061    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c1f70ee-4ca7-4550-b098-bcd450197a28-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c1f70ee-4ca7-4550-b098-bcd450197a28" (UID: "3c1f70ee-4ca7-4550-b098-bcd450197a28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:27.513362 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:27.513323    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sql2c\" (UniqueName: \"kubernetes.io/projected/3c1f70ee-4ca7-4550-b098-bcd450197a28-kube-api-access-sql2c\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:50:27.513362 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:27.513361    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c1f70ee-4ca7-4550-b098-bcd450197a28-model-cache\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:50:28.183713 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:28.183687    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-s9fqh"
Sep 16 18:50:28.183889 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:28.183686    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-s9fqh" event={"ID":"3c1f70ee-4ca7-4550-b098-bcd450197a28","Type":"ContainerDied","Data":"43a64a9c67635eb5cfc76f986163d072f2feab3c65ef70adb5e3014ba8ff8892"}
Sep 16 18:50:28.183889 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:28.183788    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43a64a9c67635eb5cfc76f986163d072f2feab3c65ef70adb5e3014ba8ff8892"
Sep 16 18:50:40.350882 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.350851    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"]
Sep 16 18:50:40.351247 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.351143    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1f70ee-4ca7-4550-b098-bcd450197a28" containerName="s3-init"
Sep 16 18:50:40.351247 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.351153    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1f70ee-4ca7-4550-b098-bcd450197a28" containerName="s3-init"
Sep 16 18:50:40.351247 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.351166    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1f70ee-4ca7-4550-b098-bcd450197a28" containerName="download-hf-model"
Sep 16 18:50:40.351247 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.351171    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1f70ee-4ca7-4550-b098-bcd450197a28" containerName="download-hf-model"
Sep 16 18:50:40.351247 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.351232    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1f70ee-4ca7-4550-b098-bcd450197a28" containerName="s3-init"
Sep 16 18:50:40.354658 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.354638    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.357816 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.357791    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 18:50:40.358126 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.358101    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 16 18:50:40.358259 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.358165    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkvz5\""
Sep 16 18:50:40.358259 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.358250    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:50:40.358423 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.358398    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:50:40.358493 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.358428    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:50:40.358493 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.358445    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:50:40.367864 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.367839    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"]
Sep 16 18:50:40.457028 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.457000    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"]
Sep 16 18:50:40.460421 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.460405    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.462921 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.462899    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 18:50:40.473205 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.473184    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"]
Sep 16 18:50:40.514320 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.514298    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18c52db8-a0e3-458f-9106-1f5bd05560a7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.514439 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.514326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76c4\" (UniqueName: \"kubernetes.io/projected/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-api-access-c76c4\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.514439 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.514371    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.514439 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.514424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.514559 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.514456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18c52db8-a0e3-458f-9106-1f5bd05560a7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.615626 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.615561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c159330-7e1e-461c-9142-c85a588def66-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.615626 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.615596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.615626 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.615617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.615850 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.615665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18c52db8-a0e3-458f-9106-1f5bd05560a7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.615850 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.615703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.615850 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.615765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c159330-7e1e-461c-9142-c85a588def66-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.615850 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.615799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18c52db8-a0e3-458f-9106-1f5bd05560a7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.616064 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.615867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76c4\" (UniqueName: \"kubernetes.io/projected/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-api-access-c76c4\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.616064 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.615911    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.616064 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.615959    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvm77\" (UniqueName: \"kubernetes.io/projected/7c159330-7e1e-461c-9142-c85a588def66-kube-api-access-bvm77\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.616179 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.616076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18c52db8-a0e3-458f-9106-1f5bd05560a7-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.616403 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.616386    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.616502 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.616482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.618337 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.618315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18c52db8-a0e3-458f-9106-1f5bd05560a7-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.624897 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.624872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76c4\" (UniqueName: \"kubernetes.io/projected/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-api-access-c76c4\") pod \"isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.667789 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.667731    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:50:40.716945 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.716909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c159330-7e1e-461c-9142-c85a588def66-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.717081 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.716981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.717081 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.717032    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvm77\" (UniqueName: \"kubernetes.io/projected/7c159330-7e1e-461c-9142-c85a588def66-kube-api-access-bvm77\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.717211 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.717086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c159330-7e1e-461c-9142-c85a588def66-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.717211 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.717158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.718259 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.717954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.718259 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.717972    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-cabundle-cert\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.718259 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:50:40.718221    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 18:50:40.718259 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.718228    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c159330-7e1e-461c-9142-c85a588def66-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.718724 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:50:40.718296    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c159330-7e1e-461c-9142-c85a588def66-proxy-tls podName:7c159330-7e1e-461c-9142-c85a588def66 nodeName:}" failed. No retries permitted until 2026-09-16 18:50:41.218276843 +0000 UTC m=+614.101099916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7c159330-7e1e-461c-9142-c85a588def66-proxy-tls") pod "isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" (UID: "7c159330-7e1e-461c-9142-c85a588def66") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 18:50:40.727851 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.727795    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvm77\" (UniqueName: \"kubernetes.io/projected/7c159330-7e1e-461c-9142-c85a588def66-kube-api-access-bvm77\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:40.943757 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:40.943715    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"]
Sep 16 18:50:40.945828 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:50:40.945794    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c52db8_a0e3_458f_9106_1f5bd05560a7.slice/crio-fb9331cba35895ae1482e5d84581eac84c2dc631444315c50e1f865a38533b27 WatchSource:0}: Error finding container fb9331cba35895ae1482e5d84581eac84c2dc631444315c50e1f865a38533b27: Status 404 returned error can't find the container with id fb9331cba35895ae1482e5d84581eac84c2dc631444315c50e1f865a38533b27
Sep 16 18:50:41.220605 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:41.220518    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c159330-7e1e-461c-9142-c85a588def66-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:41.223008 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:41.222986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c159330-7e1e-461c-9142-c85a588def66-proxy-tls\") pod \"isvc-canary-rollback-predictor-c9f4b9d7d-xv24p\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:41.239314 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:41.239285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" event={"ID":"18c52db8-a0e3-458f-9106-1f5bd05560a7","Type":"ContainerStarted","Data":"acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5"}
Sep 16 18:50:41.239461 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:41.239374    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" event={"ID":"18c52db8-a0e3-458f-9106-1f5bd05560a7","Type":"ContainerStarted","Data":"fb9331cba35895ae1482e5d84581eac84c2dc631444315c50e1f865a38533b27"}
Sep 16 18:50:41.373127 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:41.373096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:50:41.647809 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:41.647659    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"]
Sep 16 18:50:41.650814 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:50:41.650787    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c159330_7e1e_461c_9142_c85a588def66.slice/crio-bafa59da412c21475e34ab6084849af3ff9a3ea985d0bb18d54161a6a920178a WatchSource:0}: Error finding container bafa59da412c21475e34ab6084849af3ff9a3ea985d0bb18d54161a6a920178a: Status 404 returned error can't find the container with id bafa59da412c21475e34ab6084849af3ff9a3ea985d0bb18d54161a6a920178a
Sep 16 18:50:42.245384 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:42.245345    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" event={"ID":"7c159330-7e1e-461c-9142-c85a588def66","Type":"ContainerStarted","Data":"5adca3cba77cf4db3ba4f05466980685fabf3a7cd3b6ae1adc32eb1e358fc456"}
Sep 16 18:50:42.245638 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:42.245453    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" event={"ID":"7c159330-7e1e-461c-9142-c85a588def66","Type":"ContainerStarted","Data":"bafa59da412c21475e34ab6084849af3ff9a3ea985d0bb18d54161a6a920178a"}
Sep 16 18:50:45.258920 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:45.258885    2583 generic.go:356] "Generic (PLEG): container finished" podID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerID="acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5" exitCode=0
Sep 16 18:50:45.259256 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:45.258930    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" event={"ID":"18c52db8-a0e3-458f-9106-1f5bd05560a7","Type":"ContainerDied","Data":"acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5"}
Sep 16 18:50:46.269335 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:46.269188    2583 generic.go:356] "Generic (PLEG): container finished" podID="7c159330-7e1e-461c-9142-c85a588def66" containerID="5adca3cba77cf4db3ba4f05466980685fabf3a7cd3b6ae1adc32eb1e358fc456" exitCode=0
Sep 16 18:50:46.269335 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:50:46.269256    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" event={"ID":"7c159330-7e1e-461c-9142-c85a588def66","Type":"ContainerDied","Data":"5adca3cba77cf4db3ba4f05466980685fabf3a7cd3b6ae1adc32eb1e358fc456"}
Sep 16 18:51:01.376073 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:01.376036    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" event={"ID":"7c159330-7e1e-461c-9142-c85a588def66","Type":"ContainerStarted","Data":"e454e08a9e10058c0bf8338ff1c9da01602085ea979deb875a57a6997c8d079c"}
Sep 16 18:51:01.378119 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:01.378094    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" event={"ID":"18c52db8-a0e3-458f-9106-1f5bd05560a7","Type":"ContainerStarted","Data":"c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c"}
Sep 16 18:51:04.397062 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:04.397024    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" event={"ID":"7c159330-7e1e-461c-9142-c85a588def66","Type":"ContainerStarted","Data":"12d13e59aeaaacbbb4de59572a8ba051cb28c7c2841496a5c9a02cff62b749ab"}
Sep 16 18:51:04.397548 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:04.397103    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:51:04.397548 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:04.397235    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:51:04.398575 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:04.398539    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:51:04.399346 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:04.399324    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" event={"ID":"18c52db8-a0e3-458f-9106-1f5bd05560a7","Type":"ContainerStarted","Data":"96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db"}
Sep 16 18:51:04.399451 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:04.399436    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:51:04.399517 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:04.399507    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:51:04.400688 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:04.400662    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:51:04.417292 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:04.417247    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podStartSLOduration=2.946994233 podStartE2EDuration="24.417234514s" totalImagesPullingTime="17.421629994s" totalInitContainerRuntime="4.048610287s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:04.416256702 +0000 UTC m=+637.299079798" watchObservedRunningTime="2026-09-16 18:51:04.417234514 +0000 UTC m=+637.300057595"
Sep 16 18:51:04.436284 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:04.436244    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podStartSLOduration=2.315877939 podStartE2EDuration="24.436233024s" totalImagesPullingTime="18.437420484s" totalInitContainerRuntime="3.682934601s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:04.434901704 +0000 UTC m=+637.317724785" watchObservedRunningTime="2026-09-16 18:51:04.436233024 +0000 UTC m=+637.319056102"
Sep 16 18:51:05.403342 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:05.403305    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:51:05.403724 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:05.403434    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:51:10.408589 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:10.408559    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:51:10.414252 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:10.408892    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:51:10.414252 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:10.409138    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:51:10.414252 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:10.409615    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:51:20.409656 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:20.409616    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:51:20.409656 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:20.409651    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:51:30.409925 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:30.409890    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:51:30.410285 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:30.409888    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:51:40.409969 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:40.409930    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:51:40.410408 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:40.409930    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:51:50.410035 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:50.410002    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:51:50.410400 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:51:50.410000    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:52:00.409516 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:00.409480    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:52:00.409927 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:00.409600    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:52:10.409932 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:10.409896    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:52:10.410291 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:10.409896    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 16 18:52:16.654511 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:16.654483    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:52:20.410362 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:20.410291    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:52:30.172635 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:30.172596    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"]
Sep 16 18:52:30.173100 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:30.172919    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container" containerID="cri-o://c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c" gracePeriod=30
Sep 16 18:52:30.173100 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:30.172951    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kube-rbac-proxy" containerID="cri-o://96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db" gracePeriod=30
Sep 16 18:52:30.404434 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:30.404394    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 16 18:52:30.409910 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:30.409874    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 16 18:52:30.762135 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:30.762103    2583 generic.go:356] "Generic (PLEG): container finished" podID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerID="96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db" exitCode=2
Sep 16 18:52:30.762333 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:30.762185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" event={"ID":"18c52db8-a0e3-458f-9106-1f5bd05560a7","Type":"ContainerDied","Data":"96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db"}
Sep 16 18:52:35.117303 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.117276    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:52:35.212907 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.212877    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-cabundle-cert\") pod \"18c52db8-a0e3-458f-9106-1f5bd05560a7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") "
Sep 16 18:52:35.213095 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.212944    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-rbac-proxy-sar-config\") pod \"18c52db8-a0e3-458f-9106-1f5bd05560a7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") "
Sep 16 18:52:35.213095 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.212999    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18c52db8-a0e3-458f-9106-1f5bd05560a7-proxy-tls\") pod \"18c52db8-a0e3-458f-9106-1f5bd05560a7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") "
Sep 16 18:52:35.213095 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.213028    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c76c4\" (UniqueName: \"kubernetes.io/projected/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-api-access-c76c4\") pod \"18c52db8-a0e3-458f-9106-1f5bd05560a7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") "
Sep 16 18:52:35.213095 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.213093    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18c52db8-a0e3-458f-9106-1f5bd05560a7-kserve-provision-location\") pod \"18c52db8-a0e3-458f-9106-1f5bd05560a7\" (UID: \"18c52db8-a0e3-458f-9106-1f5bd05560a7\") "
Sep 16 18:52:35.213323 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.213259    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18c52db8-a0e3-458f-9106-1f5bd05560a7" (UID: "18c52db8-a0e3-458f-9106-1f5bd05560a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:35.213323 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.213289    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18c52db8-a0e3-458f-9106-1f5bd05560a7" (UID: "18c52db8-a0e3-458f-9106-1f5bd05560a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:52:35.213496 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.213475    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c52db8-a0e3-458f-9106-1f5bd05560a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18c52db8-a0e3-458f-9106-1f5bd05560a7" (UID: "18c52db8-a0e3-458f-9106-1f5bd05560a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:35.213546 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.213480    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:52:35.213546 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.213516    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:52:35.215363 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.215342    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18c52db8-a0e3-458f-9106-1f5bd05560a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18c52db8-a0e3-458f-9106-1f5bd05560a7" (UID: "18c52db8-a0e3-458f-9106-1f5bd05560a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:35.215458 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.215422    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-api-access-c76c4" (OuterVolumeSpecName: "kube-api-access-c76c4") pod "18c52db8-a0e3-458f-9106-1f5bd05560a7" (UID: "18c52db8-a0e3-458f-9106-1f5bd05560a7"). InnerVolumeSpecName "kube-api-access-c76c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:35.314840 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.314808    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18c52db8-a0e3-458f-9106-1f5bd05560a7-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:52:35.314840 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.314832    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c76c4\" (UniqueName: \"kubernetes.io/projected/18c52db8-a0e3-458f-9106-1f5bd05560a7-kube-api-access-c76c4\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:52:35.314840 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.314843    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18c52db8-a0e3-458f-9106-1f5bd05560a7-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:52:35.321869 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.321831    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"]
Sep 16 18:52:35.322206 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.322190    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kube-rbac-proxy"
Sep 16 18:52:35.322286 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.322210    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kube-rbac-proxy"
Sep 16 18:52:35.322286 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.322239    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="storage-initializer"
Sep 16 18:52:35.322286 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.322248    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="storage-initializer"
Sep 16 18:52:35.322286 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.322259    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container"
Sep 16 18:52:35.322286 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.322267    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container"
Sep 16 18:52:35.322550 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.322353    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kube-rbac-proxy"
Sep 16 18:52:35.322550 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.322367    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerName="kserve-container"
Sep 16 18:52:35.325675 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.325656    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.327712 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.327691    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c594f-predictor-serving-cert\""
Sep 16 18:52:35.327847 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.327711    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-c594f-kube-rbac-proxy-sar-config\""
Sep 16 18:52:35.338039 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.338016    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"]
Sep 16 18:52:35.473458 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.473381    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"]
Sep 16 18:52:35.476972 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.476952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.479022 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.479003    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c594f-predictor-serving-cert\""
Sep 16 18:52:35.479090 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.479042    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-c594f-kube-rbac-proxy-sar-config\""
Sep 16 18:52:35.487345 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.487321    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"]
Sep 16 18:52:35.517315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.517282    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/971a8da3-69fe-4800-ad07-a4c991615f08-proxy-tls\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.517454 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.517327    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.517454 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.517369    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.517454 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.517406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/971a8da3-69fe-4800-ad07-a4c991615f08-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.517574 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.517476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvlfn\" (UniqueName: \"kubernetes.io/projected/971a8da3-69fe-4800-ad07-a4c991615f08-kube-api-access-xvlfn\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.618137 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/971a8da3-69fe-4800-ad07-a4c991615f08-proxy-tls\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.618137 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.618396 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.618396 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/971a8da3-69fe-4800-ad07-a4c991615f08-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.618396 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.618396 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:52:35.618287    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-serving-cert: secret "isvc-sklearn-graph-raw-c594f-predictor-serving-cert" not found
Sep 16 18:52:35.618396 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba98d36-022a-41e5-b549-7092ccc5740d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.618396 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:52:35.618360    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/971a8da3-69fe-4800-ad07-a4c991615f08-proxy-tls podName:971a8da3-69fe-4800-ad07-a4c991615f08 nodeName:}" failed. No retries permitted until 2026-09-16 18:52:36.118339397 +0000 UTC m=+729.001162462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/971a8da3-69fe-4800-ad07-a4c991615f08-proxy-tls") pod "isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" (UID: "971a8da3-69fe-4800-ad07-a4c991615f08") : secret "isvc-sklearn-graph-raw-c594f-predictor-serving-cert" not found
Sep 16 18:52:35.618396 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qllxq\" (UniqueName: \"kubernetes.io/projected/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-api-access-qllxq\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.618718 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvlfn\" (UniqueName: \"kubernetes.io/projected/971a8da3-69fe-4800-ad07-a4c991615f08-kube-api-access-xvlfn\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.618718 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618483    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.618718 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba98d36-022a-41e5-b549-7092ccc5740d-proxy-tls\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.618718 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/971a8da3-69fe-4800-ad07-a4c991615f08-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.618900 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.618882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-cabundle-cert\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.619106 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.619088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.627471 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.627445    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvlfn\" (UniqueName: \"kubernetes.io/projected/971a8da3-69fe-4800-ad07-a4c991615f08-kube-api-access-xvlfn\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:35.720143 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.720105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.720320 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.720165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba98d36-022a-41e5-b549-7092ccc5740d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.720320 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.720199    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qllxq\" (UniqueName: \"kubernetes.io/projected/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-api-access-qllxq\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.720320 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.720247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.720320 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.720275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba98d36-022a-41e5-b549-7092ccc5740d-proxy-tls\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.720545 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:52:35.720422    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-serving-cert: secret "isvc-xgboost-graph-raw-c594f-predictor-serving-cert" not found
Sep 16 18:52:35.720545 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:52:35.720489    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ba98d36-022a-41e5-b549-7092ccc5740d-proxy-tls podName:4ba98d36-022a-41e5-b549-7092ccc5740d nodeName:}" failed. No retries permitted until 2026-09-16 18:52:36.22047158 +0000 UTC m=+729.103294645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ba98d36-022a-41e5-b549-7092ccc5740d-proxy-tls") pod "isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" (UID: "4ba98d36-022a-41e5-b549-7092ccc5740d") : secret "isvc-xgboost-graph-raw-c594f-predictor-serving-cert" not found
Sep 16 18:52:35.720729 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.720614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba98d36-022a-41e5-b549-7092ccc5740d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.720868 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.720848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.721026 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.721003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.729052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.729000    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qllxq\" (UniqueName: \"kubernetes.io/projected/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-api-access-qllxq\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:35.786654 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.786618    2583 generic.go:356] "Generic (PLEG): container finished" podID="18c52db8-a0e3-458f-9106-1f5bd05560a7" containerID="c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c" exitCode=0
Sep 16 18:52:35.786833 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.786691    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"
Sep 16 18:52:35.786833 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.786715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" event={"ID":"18c52db8-a0e3-458f-9106-1f5bd05560a7","Type":"ContainerDied","Data":"c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c"}
Sep 16 18:52:35.786912 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.786836    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7" event={"ID":"18c52db8-a0e3-458f-9106-1f5bd05560a7","Type":"ContainerDied","Data":"fb9331cba35895ae1482e5d84581eac84c2dc631444315c50e1f865a38533b27"}
Sep 16 18:52:35.786912 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.786865    2583 scope.go:117] "RemoveContainer" containerID="96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db"
Sep 16 18:52:35.795134 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.795115    2583 scope.go:117] "RemoveContainer" containerID="c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c"
Sep 16 18:52:35.802763 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.802637    2583 scope.go:117] "RemoveContainer" containerID="acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5"
Sep 16 18:52:35.804284 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.804263    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"]
Sep 16 18:52:35.808882 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.808855    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6dc8d44cf9-cmpm7"]
Sep 16 18:52:35.810829 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.810816    2583 scope.go:117] "RemoveContainer" containerID="96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db"
Sep 16 18:52:35.811099 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:52:35.811083    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db\": container with ID starting with 96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db not found: ID does not exist" containerID="96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db"
Sep 16 18:52:35.811155 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.811105    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db"} err="failed to get container status \"96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db\": rpc error: code = NotFound desc = could not find container \"96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db\": container with ID starting with 96f3af6de2a71c08b69fb98bf73123545083dcb503a99645dca7e20c542e06db not found: ID does not exist"
Sep 16 18:52:35.811155 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.811121    2583 scope.go:117] "RemoveContainer" containerID="c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c"
Sep 16 18:52:35.811345 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:52:35.811325    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c\": container with ID starting with c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c not found: ID does not exist" containerID="c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c"
Sep 16 18:52:35.811409 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.811356    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c"} err="failed to get container status \"c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c\": rpc error: code = NotFound desc = could not find container \"c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c\": container with ID starting with c1717b93bcf3bf58aadca2e5ac7c77c2530e0788a85d6e88724ac21bb9309b0c not found: ID does not exist"
Sep 16 18:52:35.811409 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.811380    2583 scope.go:117] "RemoveContainer" containerID="acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5"
Sep 16 18:52:35.811664 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:52:35.811648    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5\": container with ID starting with acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5 not found: ID does not exist" containerID="acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5"
Sep 16 18:52:35.811702 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:35.811668    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5"} err="failed to get container status \"acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5\": rpc error: code = NotFound desc = could not find container \"acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5\": container with ID starting with acb44ebfbd03358231f9a2e97221a678d57d702e194a5e59e38ce687eb6f13e5 not found: ID does not exist"
Sep 16 18:52:36.123616 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.123512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/971a8da3-69fe-4800-ad07-a4c991615f08-proxy-tls\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:36.126174 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.126151    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/971a8da3-69fe-4800-ad07-a4c991615f08-proxy-tls\") pod \"isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:36.224153 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.224113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba98d36-022a-41e5-b549-7092ccc5740d-proxy-tls\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:36.226668 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.226646    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba98d36-022a-41e5-b549-7092ccc5740d-proxy-tls\") pod \"isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:36.235381 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.235346    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:36.390147 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.390060    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:36.512307 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.512282    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"]
Sep 16 18:52:36.514284 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:52:36.514257    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971a8da3_69fe_4800_ad07_a4c991615f08.slice/crio-d95a4eddb6ade39f3e7a3be51815bed2939343dc321e3715062b40a7911ac476 WatchSource:0}: Error finding container d95a4eddb6ade39f3e7a3be51815bed2939343dc321e3715062b40a7911ac476: Status 404 returned error can't find the container with id d95a4eddb6ade39f3e7a3be51815bed2939343dc321e3715062b40a7911ac476
Sep 16 18:52:36.663569 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.663548    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"]
Sep 16 18:52:36.666639 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:52:36.666610    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba98d36_022a_41e5_b549_7092ccc5740d.slice/crio-8f54025636caa372b6087f8d6fb3abc8065174252d754b962873fdecd7a2f69f WatchSource:0}: Error finding container 8f54025636caa372b6087f8d6fb3abc8065174252d754b962873fdecd7a2f69f: Status 404 returned error can't find the container with id 8f54025636caa372b6087f8d6fb3abc8065174252d754b962873fdecd7a2f69f
Sep 16 18:52:36.794586 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.794537    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" event={"ID":"4ba98d36-022a-41e5-b549-7092ccc5740d","Type":"ContainerStarted","Data":"49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9"}
Sep 16 18:52:36.794812 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.794610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" event={"ID":"4ba98d36-022a-41e5-b549-7092ccc5740d","Type":"ContainerStarted","Data":"8f54025636caa372b6087f8d6fb3abc8065174252d754b962873fdecd7a2f69f"}
Sep 16 18:52:36.796404 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.796379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" event={"ID":"971a8da3-69fe-4800-ad07-a4c991615f08","Type":"ContainerStarted","Data":"48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac"}
Sep 16 18:52:36.796524 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:36.796433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" event={"ID":"971a8da3-69fe-4800-ad07-a4c991615f08","Type":"ContainerStarted","Data":"d95a4eddb6ade39f3e7a3be51815bed2939343dc321e3715062b40a7911ac476"}
Sep 16 18:52:37.659266 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:37.659234    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18c52db8-a0e3-458f-9106-1f5bd05560a7" path="/var/lib/kubelet/pods/18c52db8-a0e3-458f-9106-1f5bd05560a7/volumes"
Sep 16 18:52:40.821690 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:40.821654    2583 generic.go:356] "Generic (PLEG): container finished" podID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerID="49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9" exitCode=0
Sep 16 18:52:40.822137 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:40.821725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" event={"ID":"4ba98d36-022a-41e5-b549-7092ccc5740d","Type":"ContainerDied","Data":"49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9"}
Sep 16 18:52:40.823432 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:40.823411    2583 generic.go:356] "Generic (PLEG): container finished" podID="971a8da3-69fe-4800-ad07-a4c991615f08" containerID="48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac" exitCode=0
Sep 16 18:52:40.823516 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:40.823466    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" event={"ID":"971a8da3-69fe-4800-ad07-a4c991615f08","Type":"ContainerDied","Data":"48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac"}
Sep 16 18:52:41.832478 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:41.832439    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" event={"ID":"971a8da3-69fe-4800-ad07-a4c991615f08","Type":"ContainerStarted","Data":"7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b"}
Sep 16 18:52:41.832964 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:41.832516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" event={"ID":"971a8da3-69fe-4800-ad07-a4c991615f08","Type":"ContainerStarted","Data":"e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135"}
Sep 16 18:52:41.832964 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:41.832760    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:41.860992 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:41.860934    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podStartSLOduration=3.306155592 podStartE2EDuration="6.860911626s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.554756034s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:41.85767455 +0000 UTC m=+734.740497632" watchObservedRunningTime="2026-09-16 18:52:41.860911626 +0000 UTC m=+734.743734707"
Sep 16 18:52:42.839048 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:42.839018    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:42.840890 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:42.840857    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:52:43.845528 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:43.845489    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:52:48.851941 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:48.851883    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:52:48.852548 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:48.852481    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:52:58.852351 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:58.852306    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:52:58.920085 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:58.920057    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" event={"ID":"4ba98d36-022a-41e5-b549-7092ccc5740d","Type":"ContainerStarted","Data":"a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494"}
Sep 16 18:52:59.926950 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:59.926910    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" event={"ID":"4ba98d36-022a-41e5-b549-7092ccc5740d","Type":"ContainerStarted","Data":"55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30"}
Sep 16 18:52:59.927381 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:59.926988    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:59.927381 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:59.927107    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:52:59.928409 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:59.928388    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:52:59.947515 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:52:59.947463    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podStartSLOduration=3.0689841 podStartE2EDuration="24.947445353s" totalImagesPullingTime="17.946498872s" totalInitContainerRuntime="3.931962381s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:59.946793474 +0000 UTC m=+752.829616556" watchObservedRunningTime="2026-09-16 18:52:59.947445353 +0000 UTC m=+752.830268435"
Sep 16 18:53:00.931260 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:00.931219    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:53:05.935879 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:05.935846    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:53:05.936400 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:05.936375    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:53:08.853156 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:08.853113    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:53:15.936373 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:15.936329    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:53:18.852503 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:18.852461    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:53:25.936318 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:25.936270    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:53:28.852627 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:28.852576    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:53:35.937187 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:35.937140    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:53:38.852859 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:38.852818    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:53:45.936522 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:45.936484    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:53:48.852581 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:48.852537    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:53:51.654584 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:51.654532    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 16 18:53:55.936304 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:53:55.936264    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:54:00.281575 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:00.281545    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"]
Sep 16 18:54:00.282093 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:00.281785    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container" containerID="cri-o://e454e08a9e10058c0bf8338ff1c9da01602085ea979deb875a57a6997c8d079c" gracePeriod=30
Sep 16 18:54:00.282093 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:00.281829    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kube-rbac-proxy" containerID="cri-o://12d13e59aeaaacbbb4de59572a8ba051cb28c7c2841496a5c9a02cff62b749ab" gracePeriod=30
Sep 16 18:54:00.403978 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:00.403949    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 16 18:54:01.220805 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:01.220771    2583 generic.go:356] "Generic (PLEG): container finished" podID="7c159330-7e1e-461c-9142-c85a588def66" containerID="12d13e59aeaaacbbb4de59572a8ba051cb28c7c2841496a5c9a02cff62b749ab" exitCode=2
Sep 16 18:54:01.220983 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:01.220854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" event={"ID":"7c159330-7e1e-461c-9142-c85a588def66","Type":"ContainerDied","Data":"12d13e59aeaaacbbb4de59572a8ba051cb28c7c2841496a5c9a02cff62b749ab"}
Sep 16 18:54:01.658359 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:01.658282    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:54:04.237024 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.236995    2583 generic.go:356] "Generic (PLEG): container finished" podID="7c159330-7e1e-461c-9142-c85a588def66" containerID="e454e08a9e10058c0bf8338ff1c9da01602085ea979deb875a57a6997c8d079c" exitCode=0
Sep 16 18:54:04.237362 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.237052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" event={"ID":"7c159330-7e1e-461c-9142-c85a588def66","Type":"ContainerDied","Data":"e454e08a9e10058c0bf8338ff1c9da01602085ea979deb875a57a6997c8d079c"}
Sep 16 18:54:04.321750 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.321717    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:54:04.472649 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.472581    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-kube-rbac-proxy-sar-config\") pod \"7c159330-7e1e-461c-9142-c85a588def66\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") "
Sep 16 18:54:04.472649 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.472624    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c159330-7e1e-461c-9142-c85a588def66-proxy-tls\") pod \"7c159330-7e1e-461c-9142-c85a588def66\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") "
Sep 16 18:54:04.472861 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.472682    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-cabundle-cert\") pod \"7c159330-7e1e-461c-9142-c85a588def66\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") "
Sep 16 18:54:04.472861 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.472710    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvm77\" (UniqueName: \"kubernetes.io/projected/7c159330-7e1e-461c-9142-c85a588def66-kube-api-access-bvm77\") pod \"7c159330-7e1e-461c-9142-c85a588def66\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") "
Sep 16 18:54:04.472861 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.472767    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c159330-7e1e-461c-9142-c85a588def66-kserve-provision-location\") pod \"7c159330-7e1e-461c-9142-c85a588def66\" (UID: \"7c159330-7e1e-461c-9142-c85a588def66\") "
Sep 16 18:54:04.473138 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.473103    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c159330-7e1e-461c-9142-c85a588def66" (UID: "7c159330-7e1e-461c-9142-c85a588def66"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:04.473138 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.473114    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c159330-7e1e-461c-9142-c85a588def66-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c159330-7e1e-461c-9142-c85a588def66" (UID: "7c159330-7e1e-461c-9142-c85a588def66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:04.473138 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.473116    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c159330-7e1e-461c-9142-c85a588def66" (UID: "7c159330-7e1e-461c-9142-c85a588def66"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:04.475007 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.474981    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c159330-7e1e-461c-9142-c85a588def66-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c159330-7e1e-461c-9142-c85a588def66" (UID: "7c159330-7e1e-461c-9142-c85a588def66"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:04.475097 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.475028    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c159330-7e1e-461c-9142-c85a588def66-kube-api-access-bvm77" (OuterVolumeSpecName: "kube-api-access-bvm77") pod "7c159330-7e1e-461c-9142-c85a588def66" (UID: "7c159330-7e1e-461c-9142-c85a588def66"). InnerVolumeSpecName "kube-api-access-bvm77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:04.573340 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.573320    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:04.573340 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.573342    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c159330-7e1e-461c-9142-c85a588def66-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:04.573460 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.573351    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c159330-7e1e-461c-9142-c85a588def66-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:04.573460 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.573360    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvm77\" (UniqueName: \"kubernetes.io/projected/7c159330-7e1e-461c-9142-c85a588def66-kube-api-access-bvm77\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:04.573460 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:04.573369    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c159330-7e1e-461c-9142-c85a588def66-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:05.245608 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.245572    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"
Sep 16 18:54:05.246051 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.245564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p" event={"ID":"7c159330-7e1e-461c-9142-c85a588def66","Type":"ContainerDied","Data":"bafa59da412c21475e34ab6084849af3ff9a3ea985d0bb18d54161a6a920178a"}
Sep 16 18:54:05.246051 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.245768    2583 scope.go:117] "RemoveContainer" containerID="12d13e59aeaaacbbb4de59572a8ba051cb28c7c2841496a5c9a02cff62b749ab"
Sep 16 18:54:05.255801 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.255781    2583 scope.go:117] "RemoveContainer" containerID="e454e08a9e10058c0bf8338ff1c9da01602085ea979deb875a57a6997c8d079c"
Sep 16 18:54:05.264045 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.264027    2583 scope.go:117] "RemoveContainer" containerID="5adca3cba77cf4db3ba4f05466980685fabf3a7cd3b6ae1adc32eb1e358fc456"
Sep 16 18:54:05.268350 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.268316    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"]
Sep 16 18:54:05.269882 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.269859    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-c9f4b9d7d-xv24p"]
Sep 16 18:54:05.548145 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.548059    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"]
Sep 16 18:54:05.548407 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.548392    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container"
Sep 16 18:54:05.548466 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.548409    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container"
Sep 16 18:54:05.548466 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.548419    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kube-rbac-proxy"
Sep 16 18:54:05.548466 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.548424    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kube-rbac-proxy"
Sep 16 18:54:05.548466 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.548442    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="storage-initializer"
Sep 16 18:54:05.548466 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.548448    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="storage-initializer"
Sep 16 18:54:05.548702 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.548495    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kube-rbac-proxy"
Sep 16 18:54:05.548702 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.548503    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c159330-7e1e-461c-9142-c85a588def66" containerName="kserve-container"
Sep 16 18:54:05.553045 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.553028    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.554925 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.554905    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 18:54:05.556317 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.556302    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 18:54:05.564874 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.564854    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"]
Sep 16 18:54:05.658924 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.658900    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c159330-7e1e-461c-9142-c85a588def66" path="/var/lib/kubelet/pods/7c159330-7e1e-461c-9142-c85a588def66/volumes"
Sep 16 18:54:05.684211 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.684193    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.684319 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.684222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.684319 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.684242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8kxp\" (UniqueName: \"kubernetes.io/projected/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-api-access-m8kxp\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.684319 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.684261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.684432 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.684348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.785097 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.785064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.785209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.785114    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.785209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.785149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.785209 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.785183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kxp\" (UniqueName: \"kubernetes.io/projected/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-api-access-m8kxp\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.785363 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.785213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.785573 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.785544    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.785823 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.785806    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.785883 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.785808    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.787775 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.787721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-proxy-tls\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.794961 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.794942    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8kxp\" (UniqueName: \"kubernetes.io/projected/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-api-access-m8kxp\") pod \"isvc-canary-stop-predictor-5ccc77f5f4-hgqf6\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.863449 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.863396    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:05.937914 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:05.937891    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:54:06.141467 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:06.141446    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"]
Sep 16 18:54:06.143639 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:54:06.143607    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbefd1b_44f0_4d1c_b889_56946e3fc6bc.slice/crio-639ee73ce8a364959b71c0e8b11572b84a1c7a4397e1ced36edc9445908ac65b WatchSource:0}: Error finding container 639ee73ce8a364959b71c0e8b11572b84a1c7a4397e1ced36edc9445908ac65b: Status 404 returned error can't find the container with id 639ee73ce8a364959b71c0e8b11572b84a1c7a4397e1ced36edc9445908ac65b
Sep 16 18:54:06.250926 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:06.250896    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" event={"ID":"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc","Type":"ContainerStarted","Data":"a30805a0a576da8cdbea2a41a64933eaedd6fd39a1377a601e02f9dbbee74ebb"}
Sep 16 18:54:06.251315 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:06.250999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" event={"ID":"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc","Type":"ContainerStarted","Data":"639ee73ce8a364959b71c0e8b11572b84a1c7a4397e1ced36edc9445908ac65b"}
Sep 16 18:54:10.273883 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:10.273832    2583 generic.go:356] "Generic (PLEG): container finished" podID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerID="a30805a0a576da8cdbea2a41a64933eaedd6fd39a1377a601e02f9dbbee74ebb" exitCode=0
Sep 16 18:54:10.274340 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:10.273923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" event={"ID":"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc","Type":"ContainerDied","Data":"a30805a0a576da8cdbea2a41a64933eaedd6fd39a1377a601e02f9dbbee74ebb"}
Sep 16 18:54:11.280804 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:11.280773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" event={"ID":"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc","Type":"ContainerStarted","Data":"654f509f5cfd4552db08b33f20297e92ae010d9127daae9e391c855416c8f45b"}
Sep 16 18:54:11.281179 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:11.280833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" event={"ID":"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc","Type":"ContainerStarted","Data":"239f2ce7b0af16c9706b878dee0507a6b7283c0bc7e285954d5bdb8f5d9a14ac"}
Sep 16 18:54:11.281179 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:11.281121    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:11.281290 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:11.281269    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:11.282686 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:11.282660    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:54:11.302202 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:11.302159    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podStartSLOduration=2.660594848 podStartE2EDuration="6.302147607s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.641552759s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:11.301948401 +0000 UTC m=+824.184771483" watchObservedRunningTime="2026-09-16 18:54:11.302147607 +0000 UTC m=+824.184970698"
Sep 16 18:54:12.285782 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:12.285717    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:54:17.290067 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:17.290040    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:54:17.290519 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:17.290493    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:54:25.504955 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.504919    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"]
Sep 16 18:54:25.505414 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.505251    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container" containerID="cri-o://e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135" gracePeriod=30
Sep 16 18:54:25.505414 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.505299    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kube-rbac-proxy" containerID="cri-o://7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b" gracePeriod=30
Sep 16 18:54:25.545560 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.545537    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"]
Sep 16 18:54:25.545820 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.545794    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container" containerID="cri-o://a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494" gracePeriod=30
Sep 16 18:54:25.545933 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.545811    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kube-rbac-proxy" containerID="cri-o://55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30" gracePeriod=30
Sep 16 18:54:25.688044 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.688017    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"]
Sep 16 18:54:25.691435 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.691416    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.693352 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.693330    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-61bdc-predictor-serving-cert\""
Sep 16 18:54:25.693444 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.693331    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-61bdc-kube-rbac-proxy-sar-config\""
Sep 16 18:54:25.701347 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.701327    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"]
Sep 16 18:54:25.744380 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.744348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmfk\" (UniqueName: \"kubernetes.io/projected/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-api-access-kwmfk\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.744507 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.744464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.744567 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.744509    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c71a771f-21ec-49f0-91d8-6c05c254087f-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.744567 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.744557    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.744667 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.744591    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c71a771f-21ec-49f0-91d8-6c05c254087f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.845322 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.845250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.845322 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.845290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c71a771f-21ec-49f0-91d8-6c05c254087f-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.845490 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.845321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.845490 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.845354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c71a771f-21ec-49f0-91d8-6c05c254087f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.845490 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:54:25.845399    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-61bdc-predictor-serving-cert" not found
Sep 16 18:54:25.845490 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.845445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmfk\" (UniqueName: \"kubernetes.io/projected/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-api-access-kwmfk\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.845490 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:54:25.845473    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71a771f-21ec-49f0-91d8-6c05c254087f-proxy-tls podName:c71a771f-21ec-49f0-91d8-6c05c254087f nodeName:}" failed. No retries permitted until 2026-09-16 18:54:26.345452902 +0000 UTC m=+839.228275984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c71a771f-21ec-49f0-91d8-6c05c254087f-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" (UID: "c71a771f-21ec-49f0-91d8-6c05c254087f") : secret "isvc-xgboost-graph-raw-hpa-61bdc-predictor-serving-cert" not found
Sep 16 18:54:25.845883 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.845859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c71a771f-21ec-49f0-91d8-6c05c254087f-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.845992 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.845979    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.846046 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.846017    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.853798 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.853774    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmfk\" (UniqueName: \"kubernetes.io/projected/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-api-access-kwmfk\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:25.931550 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.931517    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 16 18:54:25.936526 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:25.936494    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 16 18:54:26.349571 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:26.349540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c71a771f-21ec-49f0-91d8-6c05c254087f-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:26.351843 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:26.351816    2583 generic.go:356] "Generic (PLEG): container finished" podID="971a8da3-69fe-4800-ad07-a4c991615f08" containerID="7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b" exitCode=2
Sep 16 18:54:26.352193 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:26.351861    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" event={"ID":"971a8da3-69fe-4800-ad07-a4c991615f08","Type":"ContainerDied","Data":"7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b"}
Sep 16 18:54:26.352418 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:26.352396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c71a771f-21ec-49f0-91d8-6c05c254087f-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:26.354201 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:26.354181    2583 generic.go:356] "Generic (PLEG): container finished" podID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerID="55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30" exitCode=2
Sep 16 18:54:26.354299 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:26.354251    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" event={"ID":"4ba98d36-022a-41e5-b549-7092ccc5740d","Type":"ContainerDied","Data":"55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30"}
Sep 16 18:54:26.602320 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:26.602232    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:26.880888 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:26.880861    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"]
Sep 16 18:54:26.883350 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:54:26.883316    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71a771f_21ec_49f0_91d8_6c05c254087f.slice/crio-0fd1031514ac28b70df71d28b0eaf6f6ee06a11817509fd4d2c1a825dfea69d6 WatchSource:0}: Error finding container 0fd1031514ac28b70df71d28b0eaf6f6ee06a11817509fd4d2c1a825dfea69d6: Status 404 returned error can't find the container with id 0fd1031514ac28b70df71d28b0eaf6f6ee06a11817509fd4d2c1a825dfea69d6
Sep 16 18:54:27.290651 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:27.290611    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:54:27.361066 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:27.361027    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" event={"ID":"c71a771f-21ec-49f0-91d8-6c05c254087f","Type":"ContainerStarted","Data":"198681f7fe6320751d50ba3dd5372022560b5247b5cac32cc72d5d878643d9ff"}
Sep 16 18:54:27.361226 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:27.361108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" event={"ID":"c71a771f-21ec-49f0-91d8-6c05c254087f","Type":"ContainerStarted","Data":"0fd1031514ac28b70df71d28b0eaf6f6ee06a11817509fd4d2c1a825dfea69d6"}
Sep 16 18:54:28.846230 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:28.846188    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 16 18:54:29.187046 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.187023    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:54:29.274972 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.274944    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-cabundle-cert\") pod \"4ba98d36-022a-41e5-b549-7092ccc5740d\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") "
Sep 16 18:54:29.274972 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.274976    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba98d36-022a-41e5-b549-7092ccc5740d-proxy-tls\") pod \"4ba98d36-022a-41e5-b549-7092ccc5740d\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") "
Sep 16 18:54:29.275134 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.275037    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba98d36-022a-41e5-b549-7092ccc5740d-kserve-provision-location\") pod \"4ba98d36-022a-41e5-b549-7092ccc5740d\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") "
Sep 16 18:54:29.275225 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.275205    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qllxq\" (UniqueName: \"kubernetes.io/projected/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-api-access-qllxq\") pod \"4ba98d36-022a-41e5-b549-7092ccc5740d\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") "
Sep 16 18:54:29.275286 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.275268    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-rbac-proxy-sar-config\") pod \"4ba98d36-022a-41e5-b549-7092ccc5740d\" (UID: \"4ba98d36-022a-41e5-b549-7092ccc5740d\") "
Sep 16 18:54:29.275343 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.275308    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ba98d36-022a-41e5-b549-7092ccc5740d" (UID: "4ba98d36-022a-41e5-b549-7092ccc5740d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:29.275390 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.275346    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba98d36-022a-41e5-b549-7092ccc5740d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ba98d36-022a-41e5-b549-7092ccc5740d" (UID: "4ba98d36-022a-41e5-b549-7092ccc5740d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:29.275520 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.275506    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:29.275573 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.275513    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ba98d36-022a-41e5-b549-7092ccc5740d" (UID: "4ba98d36-022a-41e5-b549-7092ccc5740d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:29.275573 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.275526    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba98d36-022a-41e5-b549-7092ccc5740d-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:29.277345 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.277319    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba98d36-022a-41e5-b549-7092ccc5740d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ba98d36-022a-41e5-b549-7092ccc5740d" (UID: "4ba98d36-022a-41e5-b549-7092ccc5740d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:29.277620 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.277600    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-api-access-qllxq" (OuterVolumeSpecName: "kube-api-access-qllxq") pod "4ba98d36-022a-41e5-b549-7092ccc5740d" (UID: "4ba98d36-022a-41e5-b549-7092ccc5740d"). InnerVolumeSpecName "kube-api-access-qllxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:29.372422 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.372337    2583 generic.go:356] "Generic (PLEG): container finished" podID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerID="a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494" exitCode=0
Sep 16 18:54:29.372422 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.372372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" event={"ID":"4ba98d36-022a-41e5-b549-7092ccc5740d","Type":"ContainerDied","Data":"a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494"}
Sep 16 18:54:29.372422 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.372417    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"
Sep 16 18:54:29.372627 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.372444    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k" event={"ID":"4ba98d36-022a-41e5-b549-7092ccc5740d","Type":"ContainerDied","Data":"8f54025636caa372b6087f8d6fb3abc8065174252d754b962873fdecd7a2f69f"}
Sep 16 18:54:29.372627 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.372475    2583 scope.go:117] "RemoveContainer" containerID="55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30"
Sep 16 18:54:29.376217 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.376190    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:29.376217 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.376221    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba98d36-022a-41e5-b549-7092ccc5740d-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:29.376403 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.376238    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qllxq\" (UniqueName: \"kubernetes.io/projected/4ba98d36-022a-41e5-b549-7092ccc5740d-kube-api-access-qllxq\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:29.381947 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.381930    2583 scope.go:117] "RemoveContainer" containerID="a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494"
Sep 16 18:54:29.390089 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.390072    2583 scope.go:117] "RemoveContainer" containerID="49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9"
Sep 16 18:54:29.398531 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.398511    2583 scope.go:117] "RemoveContainer" containerID="55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30"
Sep 16 18:54:29.398824 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:54:29.398800    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30\": container with ID starting with 55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30 not found: ID does not exist" containerID="55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30"
Sep 16 18:54:29.398917 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.398831    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30"} err="failed to get container status \"55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30\": rpc error: code = NotFound desc = could not find container \"55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30\": container with ID starting with 55a7673e0b8b9b0adf9957a73d47f1916d5100b4b7dbd7a98d9d2813a8df3f30 not found: ID does not exist"
Sep 16 18:54:29.398917 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.398849    2583 scope.go:117] "RemoveContainer" containerID="a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494"
Sep 16 18:54:29.399155 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:54:29.399135    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494\": container with ID starting with a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494 not found: ID does not exist" containerID="a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494"
Sep 16 18:54:29.399221 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.399161    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494"} err="failed to get container status \"a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494\": rpc error: code = NotFound desc = could not find container \"a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494\": container with ID starting with a98c4e0eda687fd3af1609b1b18428de54d3075a13ca9e53d55867f202004494 not found: ID does not exist"
Sep 16 18:54:29.399221 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.399176    2583 scope.go:117] "RemoveContainer" containerID="49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9"
Sep 16 18:54:29.399443 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:54:29.399413    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9\": container with ID starting with 49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9 not found: ID does not exist" containerID="49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9"
Sep 16 18:54:29.399513 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.399446    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9"} err="failed to get container status \"49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9\": rpc error: code = NotFound desc = could not find container \"49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9\": container with ID starting with 49fa7f344340b7a3e7c234c17dfc8c277b201ac13957b5292a048812e0bfadc9 not found: ID does not exist"
Sep 16 18:54:29.399765 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.399723    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"]
Sep 16 18:54:29.405428 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.405409    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-c594f-predictor-7db55db665-r8w7k"]
Sep 16 18:54:29.657664 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.657592    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" path="/var/lib/kubelet/pods/4ba98d36-022a-41e5-b549-7092ccc5740d/volumes"
Sep 16 18:54:29.941270 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.941252    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:54:29.980962 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.980924    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-kube-rbac-proxy-sar-config\") pod \"971a8da3-69fe-4800-ad07-a4c991615f08\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") "
Sep 16 18:54:29.981093 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.980979    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-cabundle-cert\") pod \"971a8da3-69fe-4800-ad07-a4c991615f08\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") "
Sep 16 18:54:29.981259 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.981244    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/971a8da3-69fe-4800-ad07-a4c991615f08-proxy-tls\") pod \"971a8da3-69fe-4800-ad07-a4c991615f08\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") "
Sep 16 18:54:29.981356 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.981277    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvlfn\" (UniqueName: \"kubernetes.io/projected/971a8da3-69fe-4800-ad07-a4c991615f08-kube-api-access-xvlfn\") pod \"971a8da3-69fe-4800-ad07-a4c991615f08\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") "
Sep 16 18:54:29.981356 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.981317    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/971a8da3-69fe-4800-ad07-a4c991615f08-kserve-provision-location\") pod \"971a8da3-69fe-4800-ad07-a4c991615f08\" (UID: \"971a8da3-69fe-4800-ad07-a4c991615f08\") "
Sep 16 18:54:29.981484 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.981371    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "971a8da3-69fe-4800-ad07-a4c991615f08" (UID: "971a8da3-69fe-4800-ad07-a4c991615f08"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:29.981484 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.981443    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "971a8da3-69fe-4800-ad07-a4c991615f08" (UID: "971a8da3-69fe-4800-ad07-a4c991615f08"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:54:29.981675 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.981643    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:29.981675 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.981667    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/971a8da3-69fe-4800-ad07-a4c991615f08-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:29.981979 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.981954    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971a8da3-69fe-4800-ad07-a4c991615f08-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "971a8da3-69fe-4800-ad07-a4c991615f08" (UID: "971a8da3-69fe-4800-ad07-a4c991615f08"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:29.984088 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.984063    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971a8da3-69fe-4800-ad07-a4c991615f08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "971a8da3-69fe-4800-ad07-a4c991615f08" (UID: "971a8da3-69fe-4800-ad07-a4c991615f08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:29.984383 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:29.984360    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971a8da3-69fe-4800-ad07-a4c991615f08-kube-api-access-xvlfn" (OuterVolumeSpecName: "kube-api-access-xvlfn") pod "971a8da3-69fe-4800-ad07-a4c991615f08" (UID: "971a8da3-69fe-4800-ad07-a4c991615f08"). InnerVolumeSpecName "kube-api-access-xvlfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:30.082484 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.082454    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/971a8da3-69fe-4800-ad07-a4c991615f08-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:30.082484 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.082483    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvlfn\" (UniqueName: \"kubernetes.io/projected/971a8da3-69fe-4800-ad07-a4c991615f08-kube-api-access-xvlfn\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:30.082619 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.082499    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/971a8da3-69fe-4800-ad07-a4c991615f08-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:54:30.379395 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.379318    2583 generic.go:356] "Generic (PLEG): container finished" podID="971a8da3-69fe-4800-ad07-a4c991615f08" containerID="e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135" exitCode=0
Sep 16 18:54:30.379395 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.379372    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"
Sep 16 18:54:30.379587 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.379418    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" event={"ID":"971a8da3-69fe-4800-ad07-a4c991615f08","Type":"ContainerDied","Data":"e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135"}
Sep 16 18:54:30.379587 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.379498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv" event={"ID":"971a8da3-69fe-4800-ad07-a4c991615f08","Type":"ContainerDied","Data":"d95a4eddb6ade39f3e7a3be51815bed2939343dc321e3715062b40a7911ac476"}
Sep 16 18:54:30.379587 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.379541    2583 scope.go:117] "RemoveContainer" containerID="7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b"
Sep 16 18:54:30.391062 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.391043    2583 scope.go:117] "RemoveContainer" containerID="e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135"
Sep 16 18:54:30.399095 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.399078    2583 scope.go:117] "RemoveContainer" containerID="48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac"
Sep 16 18:54:30.405121 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.405099    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"]
Sep 16 18:54:30.406884 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.406868    2583 scope.go:117] "RemoveContainer" containerID="7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b"
Sep 16 18:54:30.407182 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:54:30.407158    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b\": container with ID starting with 7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b not found: ID does not exist" containerID="7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b"
Sep 16 18:54:30.407225 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.407190    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b"} err="failed to get container status \"7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b\": rpc error: code = NotFound desc = could not find container \"7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b\": container with ID starting with 7e283b65e31a37009f414eb0279ff1061e5028eba653272e168a8509a9986a2b not found: ID does not exist"
Sep 16 18:54:30.407225 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.407208    2583 scope.go:117] "RemoveContainer" containerID="e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135"
Sep 16 18:54:30.407505 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:54:30.407488    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135\": container with ID starting with e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135 not found: ID does not exist" containerID="e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135"
Sep 16 18:54:30.407541 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.407513    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135"} err="failed to get container status \"e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135\": rpc error: code = NotFound desc = could not find container \"e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135\": container with ID starting with e80ead5302e5396795e0e1bf086c9c523f047c91d00508ae85b23d6d5e99f135 not found: ID does not exist"
Sep 16 18:54:30.407541 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.407531    2583 scope.go:117] "RemoveContainer" containerID="48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac"
Sep 16 18:54:30.407817 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:54:30.407800    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac\": container with ID starting with 48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac not found: ID does not exist" containerID="48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac"
Sep 16 18:54:30.407866 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.407823    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac"} err="failed to get container status \"48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac\": rpc error: code = NotFound desc = could not find container \"48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac\": container with ID starting with 48bf3df8179589fd3181da5b6f584b3eedc5d80d61105011da288c662b6c83ac not found: ID does not exist"
Sep 16 18:54:30.410088 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:30.410069    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-c594f-predictor-86b8c4b7d7-zclvv"]
Sep 16 18:54:31.388281 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:31.388250    2583 generic.go:356] "Generic (PLEG): container finished" podID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerID="198681f7fe6320751d50ba3dd5372022560b5247b5cac32cc72d5d878643d9ff" exitCode=0
Sep 16 18:54:31.388786 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:31.388324    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" event={"ID":"c71a771f-21ec-49f0-91d8-6c05c254087f","Type":"ContainerDied","Data":"198681f7fe6320751d50ba3dd5372022560b5247b5cac32cc72d5d878643d9ff"}
Sep 16 18:54:31.657980 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:31.657911    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" path="/var/lib/kubelet/pods/971a8da3-69fe-4800-ad07-a4c991615f08/volumes"
Sep 16 18:54:32.395625 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:32.395591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" event={"ID":"c71a771f-21ec-49f0-91d8-6c05c254087f","Type":"ContainerStarted","Data":"81d8cbc64845a1781661d08e6cc6bf8f62ee57685c464cf4da3f237d06b387e0"}
Sep 16 18:54:32.396030 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:32.395655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" event={"ID":"c71a771f-21ec-49f0-91d8-6c05c254087f","Type":"ContainerStarted","Data":"4f25346a304cf0a2fda37a5da1c751d10da46e2640e7f380464ad68a71210a51"}
Sep 16 18:54:32.396030 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:32.396000    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:32.396126 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:32.396066    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:32.397481 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:32.397449    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:54:32.423134 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:32.418796    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podStartSLOduration=3.723237596 podStartE2EDuration="7.418778461s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.695540865s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:32.416250204 +0000 UTC m=+845.299073287" watchObservedRunningTime="2026-09-16 18:54:32.418778461 +0000 UTC m=+845.301601545"
Sep 16 18:54:33.399989 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:33.399941    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:54:37.290691 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:37.290646    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:54:38.405587 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:38.405557    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:54:38.406220 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:38.406190    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:54:47.290638 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:47.290596    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:54:48.406876 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:48.406834    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:54:57.291380 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:57.291336    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:54:58.406825 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:54:58.406787    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:55:07.290467 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:07.290422    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:55:08.406134 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:08.406096    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:55:17.290894 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:17.290847    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:55:18.406467 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:18.406424    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:55:27.291263 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:27.291232    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:55:28.406943 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:28.406912    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:55:35.527534 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:35.527500    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"]
Sep 16 18:55:35.527930 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:35.527716    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container" containerID="cri-o://239f2ce7b0af16c9706b878dee0507a6b7283c0bc7e285954d5bdb8f5d9a14ac" gracePeriod=30
Sep 16 18:55:35.527930 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:35.527778    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kube-rbac-proxy" containerID="cri-o://654f509f5cfd4552db08b33f20297e92ae010d9127daae9e391c855416c8f45b" gracePeriod=30
Sep 16 18:55:35.685562 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:35.685532    2583 generic.go:356] "Generic (PLEG): container finished" podID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerID="654f509f5cfd4552db08b33f20297e92ae010d9127daae9e391c855416c8f45b" exitCode=2
Sep 16 18:55:35.685674 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:35.685588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" event={"ID":"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc","Type":"ContainerDied","Data":"654f509f5cfd4552db08b33f20297e92ae010d9127daae9e391c855416c8f45b"}
Sep 16 18:55:37.286121 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:37.286083    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 16 18:55:37.290419 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:37.290394    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 16 18:55:39.704287 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.704244    2583 generic.go:356] "Generic (PLEG): container finished" podID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerID="239f2ce7b0af16c9706b878dee0507a6b7283c0bc7e285954d5bdb8f5d9a14ac" exitCode=0
Sep 16 18:55:39.704818 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.704301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" event={"ID":"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc","Type":"ContainerDied","Data":"239f2ce7b0af16c9706b878dee0507a6b7283c0bc7e285954d5bdb8f5d9a14ac"}
Sep 16 18:55:39.764443 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.764419    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:55:39.840123 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.840049    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kserve-provision-location\") pod \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") "
Sep 16 18:55:39.840123 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.840079    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-cabundle-cert\") pod \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") "
Sep 16 18:55:39.840331 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.840138    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-rbac-proxy-sar-config\") pod \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") "
Sep 16 18:55:39.840331 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.840159    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8kxp\" (UniqueName: \"kubernetes.io/projected/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-api-access-m8kxp\") pod \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") "
Sep 16 18:55:39.840331 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.840199    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-proxy-tls\") pod \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\" (UID: \"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc\") "
Sep 16 18:55:39.840489 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.840372    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" (UID: "2cbefd1b-44f0-4d1c-b889-56946e3fc6bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:39.840540 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.840496    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" (UID: "2cbefd1b-44f0-4d1c-b889-56946e3fc6bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:39.840540 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.840515    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" (UID: "2cbefd1b-44f0-4d1c-b889-56946e3fc6bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:39.842581 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.842541    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" (UID: "2cbefd1b-44f0-4d1c-b889-56946e3fc6bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:39.842698 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.842644    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-api-access-m8kxp" (OuterVolumeSpecName: "kube-api-access-m8kxp") pod "2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" (UID: "2cbefd1b-44f0-4d1c-b889-56946e3fc6bc"). InnerVolumeSpecName "kube-api-access-m8kxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:39.941491 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.941454    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.941491 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.941488    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.941653 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.941503    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.941653 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.941516    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8kxp\" (UniqueName: \"kubernetes.io/projected/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-kube-api-access-m8kxp\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:55:39.941653 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:39.941528    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:55:40.711177 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:40.711149    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"
Sep 16 18:55:40.711546 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:40.711146    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6" event={"ID":"2cbefd1b-44f0-4d1c-b889-56946e3fc6bc","Type":"ContainerDied","Data":"639ee73ce8a364959b71c0e8b11572b84a1c7a4397e1ced36edc9445908ac65b"}
Sep 16 18:55:40.711546 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:40.711281    2583 scope.go:117] "RemoveContainer" containerID="654f509f5cfd4552db08b33f20297e92ae010d9127daae9e391c855416c8f45b"
Sep 16 18:55:40.719978 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:40.719961    2583 scope.go:117] "RemoveContainer" containerID="239f2ce7b0af16c9706b878dee0507a6b7283c0bc7e285954d5bdb8f5d9a14ac"
Sep 16 18:55:40.727477 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:40.727460    2583 scope.go:117] "RemoveContainer" containerID="a30805a0a576da8cdbea2a41a64933eaedd6fd39a1377a601e02f9dbbee74ebb"
Sep 16 18:55:40.735540 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:40.735520    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"]
Sep 16 18:55:40.742328 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:40.742309    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ccc77f5f4-hgqf6"]
Sep 16 18:55:41.658635 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:55:41.658597    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" path="/var/lib/kubelet/pods/2cbefd1b-44f0-4d1c-b889-56946e3fc6bc/volumes"
Sep 16 18:56:15.915507 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:15.915478    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"]
Sep 16 18:56:15.918050 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:15.915726    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kube-rbac-proxy" containerID="cri-o://81d8cbc64845a1781661d08e6cc6bf8f62ee57685c464cf4da3f237d06b387e0" gracePeriod=30
Sep 16 18:56:15.918050 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:15.915710    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container" containerID="cri-o://4f25346a304cf0a2fda37a5da1c751d10da46e2640e7f380464ad68a71210a51" gracePeriod=30
Sep 16 18:56:16.860584 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:16.860552    2583 generic.go:356] "Generic (PLEG): container finished" podID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerID="81d8cbc64845a1781661d08e6cc6bf8f62ee57685c464cf4da3f237d06b387e0" exitCode=2
Sep 16 18:56:16.860756 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:16.860619    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" event={"ID":"c71a771f-21ec-49f0-91d8-6c05c254087f","Type":"ContainerDied","Data":"81d8cbc64845a1781661d08e6cc6bf8f62ee57685c464cf4da3f237d06b387e0"}
Sep 16 18:56:18.400914 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:18.400870    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 16 18:56:18.407067 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:18.407043    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 16 18:56:18.870279 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:18.870246    2583 generic.go:356] "Generic (PLEG): container finished" podID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerID="4f25346a304cf0a2fda37a5da1c751d10da46e2640e7f380464ad68a71210a51" exitCode=0
Sep 16 18:56:18.870464 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:18.870304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" event={"ID":"c71a771f-21ec-49f0-91d8-6c05c254087f","Type":"ContainerDied","Data":"4f25346a304cf0a2fda37a5da1c751d10da46e2640e7f380464ad68a71210a51"}
Sep 16 18:56:18.953077 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:18.953058    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:56:19.149410 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.149345    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwmfk\" (UniqueName: \"kubernetes.io/projected/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-api-access-kwmfk\") pod \"c71a771f-21ec-49f0-91d8-6c05c254087f\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") "
Sep 16 18:56:19.149410 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.149382    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-rbac-proxy-sar-config\") pod \"c71a771f-21ec-49f0-91d8-6c05c254087f\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") "
Sep 16 18:56:19.149620 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.149434    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c71a771f-21ec-49f0-91d8-6c05c254087f-kserve-provision-location\") pod \"c71a771f-21ec-49f0-91d8-6c05c254087f\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") "
Sep 16 18:56:19.149620 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.149461    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-cabundle-cert\") pod \"c71a771f-21ec-49f0-91d8-6c05c254087f\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") "
Sep 16 18:56:19.149620 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.149478    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c71a771f-21ec-49f0-91d8-6c05c254087f-proxy-tls\") pod \"c71a771f-21ec-49f0-91d8-6c05c254087f\" (UID: \"c71a771f-21ec-49f0-91d8-6c05c254087f\") "
Sep 16 18:56:19.149807 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.149726    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71a771f-21ec-49f0-91d8-6c05c254087f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c71a771f-21ec-49f0-91d8-6c05c254087f" (UID: "c71a771f-21ec-49f0-91d8-6c05c254087f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:19.149873 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.149852    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c71a771f-21ec-49f0-91d8-6c05c254087f" (UID: "c71a771f-21ec-49f0-91d8-6c05c254087f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:19.149910 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.149850    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c71a771f-21ec-49f0-91d8-6c05c254087f" (UID: "c71a771f-21ec-49f0-91d8-6c05c254087f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:19.151724 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.151694    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-api-access-kwmfk" (OuterVolumeSpecName: "kube-api-access-kwmfk") pod "c71a771f-21ec-49f0-91d8-6c05c254087f" (UID: "c71a771f-21ec-49f0-91d8-6c05c254087f"). InnerVolumeSpecName "kube-api-access-kwmfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:19.151845 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.151699    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71a771f-21ec-49f0-91d8-6c05c254087f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c71a771f-21ec-49f0-91d8-6c05c254087f" (UID: "c71a771f-21ec-49f0-91d8-6c05c254087f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:19.250053 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.250027    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:56:19.250053 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.250049    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c71a771f-21ec-49f0-91d8-6c05c254087f-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:56:19.250193 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.250058    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwmfk\" (UniqueName: \"kubernetes.io/projected/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-api-access-kwmfk\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:56:19.250193 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.250067    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c71a771f-21ec-49f0-91d8-6c05c254087f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:56:19.250193 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.250077    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c71a771f-21ec-49f0-91d8-6c05c254087f-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:56:19.877695 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.877655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875" event={"ID":"c71a771f-21ec-49f0-91d8-6c05c254087f","Type":"ContainerDied","Data":"0fd1031514ac28b70df71d28b0eaf6f6ee06a11817509fd4d2c1a825dfea69d6"}
Sep 16 18:56:19.878076 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.877724    2583 scope.go:117] "RemoveContainer" containerID="81d8cbc64845a1781661d08e6cc6bf8f62ee57685c464cf4da3f237d06b387e0"
Sep 16 18:56:19.878076 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.877671    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"
Sep 16 18:56:19.887704 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.887685    2583 scope.go:117] "RemoveContainer" containerID="4f25346a304cf0a2fda37a5da1c751d10da46e2640e7f380464ad68a71210a51"
Sep 16 18:56:19.895873 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.895848    2583 scope.go:117] "RemoveContainer" containerID="198681f7fe6320751d50ba3dd5372022560b5247b5cac32cc72d5d878643d9ff"
Sep 16 18:56:19.900622 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.900600    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"]
Sep 16 18:56:19.905512 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:19.905490    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-61bdc-predictor-b985c5757-pf875"]
Sep 16 18:56:21.658317 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:56:21.658274    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" path="/var/lib/kubelet/pods/c71a771f-21ec-49f0-91d8-6c05c254087f/volumes"
Sep 16 18:57:30.991646 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.991617    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"]
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.991943    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.991957    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.991965    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="storage-initializer"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.991970    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="storage-initializer"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.991981    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.991987    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.991996    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992004    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992012    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992017    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992023    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="storage-initializer"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992028    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="storage-initializer"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992035    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992039    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992046    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="storage-initializer"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992050    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="storage-initializer"
Sep 16 18:57:30.992052 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992057    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992062    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992075    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992082    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992089    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="storage-initializer"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992093    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="storage-initializer"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992104    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992109    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992158    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992166    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992171    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992177    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kube-rbac-proxy"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992183    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba98d36-022a-41e5-b549-7092ccc5740d" containerName="kserve-container"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992189    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cbefd1b-44f0-4d1c-b889-56946e3fc6bc" containerName="kserve-container"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992195    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71a771f-21ec-49f0-91d8-6c05c254087f" containerName="kserve-container"
Sep 16 18:57:30.992648 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:30.992201    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="971a8da3-69fe-4800-ad07-a4c991615f08" containerName="kserve-container"
Sep 16 18:57:31.002409 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.002384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.004463 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.004430    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-7f5d46\""
Sep 16 18:57:31.004597 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.004463    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:57:31.004662 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.004597    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-7f5d46-dockercfg-xnk6p\""
Sep 16 18:57:31.004662 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.004628    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:57:31.004662 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.004597    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-7f5d46-predictor-serving-cert\""
Sep 16 18:57:31.004835 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.004819    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:57:31.004901 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.004885    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-7f5d46-kube-rbac-proxy-sar-config\""
Sep 16 18:57:31.008372 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.008350    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"]
Sep 16 18:57:31.021684 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.021601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.021865 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.021833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abd05f08-5e2b-4b73-a208-13eabd774f24-kserve-provision-location\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.022106 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.021920    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr6nd\" (UniqueName: \"kubernetes.io/projected/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-api-access-vr6nd\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.022106 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.021956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-cabundle-cert\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.022106 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.021995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.122533 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.122512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abd05f08-5e2b-4b73-a208-13eabd774f24-kserve-provision-location\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.122646 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.122549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr6nd\" (UniqueName: \"kubernetes.io/projected/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-api-access-vr6nd\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.122646 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.122570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-cabundle-cert\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.122646 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.122590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.122646 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.122632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.122911 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:57:31.122732    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-serving-cert: secret "isvc-secondary-7f5d46-predictor-serving-cert" not found
Sep 16 18:57:31.122911 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:57:31.122818    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls podName:abd05f08-5e2b-4b73-a208-13eabd774f24 nodeName:}" failed. No retries permitted until 2026-09-16 18:57:31.622799967 +0000 UTC m=+1024.505623029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls") pod "isvc-secondary-7f5d46-predictor-66cd696585-j6fj6" (UID: "abd05f08-5e2b-4b73-a208-13eabd774f24") : secret "isvc-secondary-7f5d46-predictor-serving-cert" not found
Sep 16 18:57:31.123268 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.123248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.123365 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.123247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abd05f08-5e2b-4b73-a208-13eabd774f24-kserve-provision-location\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.123365 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.123321    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-cabundle-cert\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.134856 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.134829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr6nd\" (UniqueName: \"kubernetes.io/projected/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-api-access-vr6nd\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.625836 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:31.625804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:31.625952 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:57:31.625936    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-serving-cert: secret "isvc-secondary-7f5d46-predictor-serving-cert" not found
Sep 16 18:57:31.626008 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:57:31.625999    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls podName:abd05f08-5e2b-4b73-a208-13eabd774f24 nodeName:}" failed. No retries permitted until 2026-09-16 18:57:32.625984728 +0000 UTC m=+1025.508807787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls") pod "isvc-secondary-7f5d46-predictor-66cd696585-j6fj6" (UID: "abd05f08-5e2b-4b73-a208-13eabd774f24") : secret "isvc-secondary-7f5d46-predictor-serving-cert" not found
Sep 16 18:57:32.633085 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:32.633050    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:32.635788 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:32.635765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls\") pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") " pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:32.813867 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:32.813839    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:33.091594 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:33.091564    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"]
Sep 16 18:57:33.093587 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:57:33.093557    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd05f08_5e2b_4b73_a208_13eabd774f24.slice/crio-183e6806a2116fa00639a1d0f312045f550ea6f7e851f0524d17352186a03275 WatchSource:0}: Error finding container 183e6806a2116fa00639a1d0f312045f550ea6f7e851f0524d17352186a03275: Status 404 returned error can't find the container with id 183e6806a2116fa00639a1d0f312045f550ea6f7e851f0524d17352186a03275
Sep 16 18:57:33.179488 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:33.179455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6" event={"ID":"abd05f08-5e2b-4b73-a208-13eabd774f24","Type":"ContainerStarted","Data":"f1999bf4c24842f02a3392b5ccd84773722a5cd9c74489dae84fb27a496039b2"}
Sep 16 18:57:33.179609 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:33.179522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6" event={"ID":"abd05f08-5e2b-4b73-a208-13eabd774f24","Type":"ContainerStarted","Data":"183e6806a2116fa00639a1d0f312045f550ea6f7e851f0524d17352186a03275"}
Sep 16 18:57:38.203163 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:38.203134    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7f5d46-predictor-66cd696585-j6fj6_abd05f08-5e2b-4b73-a208-13eabd774f24/storage-initializer/0.log"
Sep 16 18:57:38.203517 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:38.203177    2583 generic.go:356] "Generic (PLEG): container finished" podID="abd05f08-5e2b-4b73-a208-13eabd774f24" containerID="f1999bf4c24842f02a3392b5ccd84773722a5cd9c74489dae84fb27a496039b2" exitCode=1
Sep 16 18:57:38.203517 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:38.203237    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6" event={"ID":"abd05f08-5e2b-4b73-a208-13eabd774f24","Type":"ContainerDied","Data":"f1999bf4c24842f02a3392b5ccd84773722a5cd9c74489dae84fb27a496039b2"}
Sep 16 18:57:39.210008 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:39.209978    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7f5d46-predictor-66cd696585-j6fj6_abd05f08-5e2b-4b73-a208-13eabd774f24/storage-initializer/0.log"
Sep 16 18:57:39.210394 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:39.210098    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6" event={"ID":"abd05f08-5e2b-4b73-a208-13eabd774f24","Type":"ContainerStarted","Data":"676a58836b6c197bbbc185b189cf29a58076a630c1761c58e2817189e26563cd"}
Sep 16 18:57:44.234262 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:44.234228    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7f5d46-predictor-66cd696585-j6fj6_abd05f08-5e2b-4b73-a208-13eabd774f24/storage-initializer/1.log"
Sep 16 18:57:44.234706 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:44.234688    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7f5d46-predictor-66cd696585-j6fj6_abd05f08-5e2b-4b73-a208-13eabd774f24/storage-initializer/0.log"
Sep 16 18:57:44.234807 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:44.234725    2583 generic.go:356] "Generic (PLEG): container finished" podID="abd05f08-5e2b-4b73-a208-13eabd774f24" containerID="676a58836b6c197bbbc185b189cf29a58076a630c1761c58e2817189e26563cd" exitCode=1
Sep 16 18:57:44.234807 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:44.234785    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6" event={"ID":"abd05f08-5e2b-4b73-a208-13eabd774f24","Type":"ContainerDied","Data":"676a58836b6c197bbbc185b189cf29a58076a630c1761c58e2817189e26563cd"}
Sep 16 18:57:44.234882 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:44.234845    2583 scope.go:117] "RemoveContainer" containerID="f1999bf4c24842f02a3392b5ccd84773722a5cd9c74489dae84fb27a496039b2"
Sep 16 18:57:44.235172 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:44.235146    2583 scope.go:117] "RemoveContainer" containerID="f1999bf4c24842f02a3392b5ccd84773722a5cd9c74489dae84fb27a496039b2"
Sep 16 18:57:44.249373 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:57:44.249323    2583 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-7f5d46-predictor-66cd696585-j6fj6_kserve-ci-e2e-test_abd05f08-5e2b-4b73-a208-13eabd774f24_0 in pod sandbox 183e6806a2116fa00639a1d0f312045f550ea6f7e851f0524d17352186a03275 from index: no such id: 'f1999bf4c24842f02a3392b5ccd84773722a5cd9c74489dae84fb27a496039b2'" containerID="f1999bf4c24842f02a3392b5ccd84773722a5cd9c74489dae84fb27a496039b2"
Sep 16 18:57:44.249537 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:57:44.249477    2583 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-7f5d46-predictor-66cd696585-j6fj6_kserve-ci-e2e-test_abd05f08-5e2b-4b73-a208-13eabd774f24_0 in pod sandbox 183e6806a2116fa00639a1d0f312045f550ea6f7e851f0524d17352186a03275 from index: no such id: 'f1999bf4c24842f02a3392b5ccd84773722a5cd9c74489dae84fb27a496039b2'; Skipping pod \"isvc-secondary-7f5d46-predictor-66cd696585-j6fj6_kserve-ci-e2e-test(abd05f08-5e2b-4b73-a208-13eabd774f24)\"" logger="UnhandledError"
Sep 16 18:57:44.249787 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:57:44.249733    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-7f5d46-predictor-66cd696585-j6fj6_kserve-ci-e2e-test(abd05f08-5e2b-4b73-a208-13eabd774f24)\"" pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6" podUID="abd05f08-5e2b-4b73-a208-13eabd774f24"
Sep 16 18:57:45.241053 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:45.241025    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7f5d46-predictor-66cd696585-j6fj6_abd05f08-5e2b-4b73-a208-13eabd774f24/storage-initializer/1.log"
Sep 16 18:57:49.007088 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.007051    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"]
Sep 16 18:57:49.137363 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.137343    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7f5d46-predictor-66cd696585-j6fj6_abd05f08-5e2b-4b73-a208-13eabd774f24/storage-initializer/1.log"
Sep 16 18:57:49.140873 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.140856    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:49.258036 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.257977    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr6nd\" (UniqueName: \"kubernetes.io/projected/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-api-access-vr6nd\") pod \"abd05f08-5e2b-4b73-a208-13eabd774f24\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") "
Sep 16 18:57:49.258036 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.258015    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-rbac-proxy-sar-config\") pod \"abd05f08-5e2b-4b73-a208-13eabd774f24\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") "
Sep 16 18:57:49.258196 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.258073    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abd05f08-5e2b-4b73-a208-13eabd774f24-kserve-provision-location\") pod \"abd05f08-5e2b-4b73-a208-13eabd774f24\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") "
Sep 16 18:57:49.258196 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.258121    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls\") pod \"abd05f08-5e2b-4b73-a208-13eabd774f24\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") "
Sep 16 18:57:49.258196 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.258172    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-cabundle-cert\") pod \"abd05f08-5e2b-4b73-a208-13eabd774f24\" (UID: \"abd05f08-5e2b-4b73-a208-13eabd774f24\") "
Sep 16 18:57:49.258408 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.258377    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd05f08-5e2b-4b73-a208-13eabd774f24-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abd05f08-5e2b-4b73-a208-13eabd774f24" (UID: "abd05f08-5e2b-4b73-a208-13eabd774f24"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:49.258478 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.258412    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "abd05f08-5e2b-4b73-a208-13eabd774f24" (UID: "abd05f08-5e2b-4b73-a208-13eabd774f24"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:49.258626 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.258603    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "abd05f08-5e2b-4b73-a208-13eabd774f24" (UID: "abd05f08-5e2b-4b73-a208-13eabd774f24"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:57:49.259206 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.259189    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-7f5d46-predictor-66cd696585-j6fj6_abd05f08-5e2b-4b73-a208-13eabd774f24/storage-initializer/1.log"
Sep 16 18:57:49.259293 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.259233    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6" event={"ID":"abd05f08-5e2b-4b73-a208-13eabd774f24","Type":"ContainerDied","Data":"183e6806a2116fa00639a1d0f312045f550ea6f7e851f0524d17352186a03275"}
Sep 16 18:57:49.259293 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.259290    2583 scope.go:117] "RemoveContainer" containerID="676a58836b6c197bbbc185b189cf29a58076a630c1761c58e2817189e26563cd"
Sep 16 18:57:49.259373 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.259289    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"
Sep 16 18:57:49.260360 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.260337    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-api-access-vr6nd" (OuterVolumeSpecName: "kube-api-access-vr6nd") pod "abd05f08-5e2b-4b73-a208-13eabd774f24" (UID: "abd05f08-5e2b-4b73-a208-13eabd774f24"). InnerVolumeSpecName "kube-api-access-vr6nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:49.260540 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.260518    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abd05f08-5e2b-4b73-a208-13eabd774f24" (UID: "abd05f08-5e2b-4b73-a208-13eabd774f24"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:49.359172 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.359148    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abd05f08-5e2b-4b73-a208-13eabd774f24-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:57:49.359261 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.359173    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abd05f08-5e2b-4b73-a208-13eabd774f24-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:57:49.359261 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.359183    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:57:49.359261 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.359192    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr6nd\" (UniqueName: \"kubernetes.io/projected/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-api-access-vr6nd\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:57:49.359261 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.359200    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abd05f08-5e2b-4b73-a208-13eabd774f24-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:57:49.599850 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.599818    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"]
Sep 16 18:57:49.603903 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.603877    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-7f5d46-predictor-66cd696585-j6fj6"]
Sep 16 18:57:49.659396 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:49.659366    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abd05f08-5e2b-4b73-a208-13eabd774f24" path="/var/lib/kubelet/pods/abd05f08-5e2b-4b73-a208-13eabd774f24/volumes"
Sep 16 18:57:54.240625 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.240592    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"]
Sep 16 18:57:54.241036 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.240918    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd05f08-5e2b-4b73-a208-13eabd774f24" containerName="storage-initializer"
Sep 16 18:57:54.241036 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.240931    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd05f08-5e2b-4b73-a208-13eabd774f24" containerName="storage-initializer"
Sep 16 18:57:54.241036 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.240944    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd05f08-5e2b-4b73-a208-13eabd774f24" containerName="storage-initializer"
Sep 16 18:57:54.241036 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.240949    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd05f08-5e2b-4b73-a208-13eabd774f24" containerName="storage-initializer"
Sep 16 18:57:54.241036 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.240996    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abd05f08-5e2b-4b73-a208-13eabd774f24" containerName="storage-initializer"
Sep 16 18:57:54.241036 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.241003    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abd05f08-5e2b-4b73-a208-13eabd774f24" containerName="storage-initializer"
Sep 16 18:57:54.243987 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.243971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.250836 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.250813    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-ee84dd-predictor-serving-cert\""
Sep 16 18:57:54.250977 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.250897    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:57:54.250977 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.250898    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-ee84dd-dockercfg-h2hcc\""
Sep 16 18:57:54.251101 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.250823    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-ee84dd\""
Sep 16 18:57:54.251386 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.251368    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-ee84dd-kube-rbac-proxy-sar-config\""
Sep 16 18:57:54.251466 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.251441    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:57:54.251524 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.251465    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:57:54.254836 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.254813    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"]
Sep 16 18:57:54.297607 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.297584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff58008-28af-4c08-8050-5960ac71f4cf-proxy-tls\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.297706 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.297610    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.297706 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.297634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ff58008-28af-4c08-8050-5960ac71f4cf-kserve-provision-location\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.297706 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.297683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55d7\" (UniqueName: \"kubernetes.io/projected/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-api-access-f55d7\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.297881 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.297786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-cabundle-cert\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.398850 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.398826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff58008-28af-4c08-8050-5960ac71f4cf-proxy-tls\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.398957 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.398859    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.398957 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.398893    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ff58008-28af-4c08-8050-5960ac71f4cf-kserve-provision-location\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.398957 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.398933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55d7\" (UniqueName: \"kubernetes.io/projected/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-api-access-f55d7\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.399119 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:57:54.398959    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-serving-cert: secret "isvc-init-fail-ee84dd-predictor-serving-cert" not found
Sep 16 18:57:54.399119 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.398989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-cabundle-cert\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.399119 ip-10-0-131-137 kubenswrapper[2583]: E0916 18:57:54.399026    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ff58008-28af-4c08-8050-5960ac71f4cf-proxy-tls podName:3ff58008-28af-4c08-8050-5960ac71f4cf nodeName:}" failed. No retries permitted until 2026-09-16 18:57:54.899006401 +0000 UTC m=+1047.781829477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3ff58008-28af-4c08-8050-5960ac71f4cf-proxy-tls") pod "isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv" (UID: "3ff58008-28af-4c08-8050-5960ac71f4cf") : secret "isvc-init-fail-ee84dd-predictor-serving-cert" not found
Sep 16 18:57:54.399308 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.399286    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ff58008-28af-4c08-8050-5960ac71f4cf-kserve-provision-location\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.399563 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.399538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.399601 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.399570    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-cabundle-cert\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.409151 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.409125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55d7\" (UniqueName: \"kubernetes.io/projected/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-api-access-f55d7\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.903638 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.903613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff58008-28af-4c08-8050-5960ac71f4cf-proxy-tls\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:54.906403 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:54.906382    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff58008-28af-4c08-8050-5960ac71f4cf-proxy-tls\") pod \"isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") " pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:55.161804 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:55.161716    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:57:55.435162 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:55.435139    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"]
Sep 16 18:57:55.437318 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:57:55.437292    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff58008_28af_4c08_8050_5960ac71f4cf.slice/crio-c5bdfcc5fa0e288ecd8cac3bc788b3101925c7d32cceb50e4ffe46689c3f8a7a WatchSource:0}: Error finding container c5bdfcc5fa0e288ecd8cac3bc788b3101925c7d32cceb50e4ffe46689c3f8a7a: Status 404 returned error can't find the container with id c5bdfcc5fa0e288ecd8cac3bc788b3101925c7d32cceb50e4ffe46689c3f8a7a
Sep 16 18:57:56.290599 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:56.290565    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv" event={"ID":"3ff58008-28af-4c08-8050-5960ac71f4cf","Type":"ContainerStarted","Data":"114d98076caf14928840f433255c9e8d709781b6b86141c00f0c84d9037433f5"}
Sep 16 18:57:56.290841 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:57:56.290622    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv" event={"ID":"3ff58008-28af-4c08-8050-5960ac71f4cf","Type":"ContainerStarted","Data":"c5bdfcc5fa0e288ecd8cac3bc788b3101925c7d32cceb50e4ffe46689c3f8a7a"}
Sep 16 18:58:00.308048 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:00.308021    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv_3ff58008-28af-4c08-8050-5960ac71f4cf/storage-initializer/0.log"
Sep 16 18:58:00.308476 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:00.308059    2583 generic.go:356] "Generic (PLEG): container finished" podID="3ff58008-28af-4c08-8050-5960ac71f4cf" containerID="114d98076caf14928840f433255c9e8d709781b6b86141c00f0c84d9037433f5" exitCode=1
Sep 16 18:58:00.308476 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:00.308135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv" event={"ID":"3ff58008-28af-4c08-8050-5960ac71f4cf","Type":"ContainerDied","Data":"114d98076caf14928840f433255c9e8d709781b6b86141c00f0c84d9037433f5"}
Sep 16 18:58:01.313993 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.313968    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv_3ff58008-28af-4c08-8050-5960ac71f4cf/storage-initializer/0.log"
Sep 16 18:58:01.314361 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.314059    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv" event={"ID":"3ff58008-28af-4c08-8050-5960ac71f4cf","Type":"ContainerStarted","Data":"a7b80451ba9f35d3e6b76ef7f21337291fed01b350b85ec426b3c94f015f9345"}
Sep 16 18:58:01.343016 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.342985    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"]
Sep 16 18:58:01.345974 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.345959    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.348307 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.348288    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkvz5\""
Sep 16 18:58:01.348614 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.348587    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-cf1e0-predictor-serving-cert\""
Sep 16 18:58:01.348784 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.348769    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-cf1e0-kube-rbac-proxy-sar-config\""
Sep 16 18:58:01.349017 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.348998    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:58:01.383786 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.383764    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"]
Sep 16 18:58:01.460556 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.460531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-cabundle-cert\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.460672 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.460646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3078bb75-1754-4187-8593-143f47243046-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.460795 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.460773    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3078bb75-1754-4187-8593-143f47243046-proxy-tls\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.460927 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.460808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5j9\" (UniqueName: \"kubernetes.io/projected/3078bb75-1754-4187-8593-143f47243046-kube-api-access-vq5j9\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.460927 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.460856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.561833 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.561804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3078bb75-1754-4187-8593-143f47243046-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.561969 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.561878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3078bb75-1754-4187-8593-143f47243046-proxy-tls\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.561969 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.561897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5j9\" (UniqueName: \"kubernetes.io/projected/3078bb75-1754-4187-8593-143f47243046-kube-api-access-vq5j9\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.561969 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.561932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.561969 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.561950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-cabundle-cert\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.562301 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.562276    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3078bb75-1754-4187-8593-143f47243046-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.562548 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.562527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-cabundle-cert\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.562594 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.562572    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.564638 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.564595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3078bb75-1754-4187-8593-143f47243046-proxy-tls\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.571637 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.571619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5j9\" (UniqueName: \"kubernetes.io/projected/3078bb75-1754-4187-8593-143f47243046-kube-api-access-vq5j9\") pod \"isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.655389 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.655363    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:01.929178 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:01.929154    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"]
Sep 16 18:58:01.931586 ip-10-0-131-137 kubenswrapper[2583]: W0916 18:58:01.931562    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3078bb75_1754_4187_8593_143f47243046.slice/crio-59cb3238ad7715383be0da40c410514bcff898e318bb0201ca71e50500593fc7 WatchSource:0}: Error finding container 59cb3238ad7715383be0da40c410514bcff898e318bb0201ca71e50500593fc7: Status 404 returned error can't find the container with id 59cb3238ad7715383be0da40c410514bcff898e318bb0201ca71e50500593fc7
Sep 16 18:58:02.323339 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:02.323257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" event={"ID":"3078bb75-1754-4187-8593-143f47243046","Type":"ContainerStarted","Data":"e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4"}
Sep 16 18:58:02.323339 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:02.323312    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" event={"ID":"3078bb75-1754-4187-8593-143f47243046","Type":"ContainerStarted","Data":"59cb3238ad7715383be0da40c410514bcff898e318bb0201ca71e50500593fc7"}
Sep 16 18:58:04.308172 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:04.308139    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"]
Sep 16 18:58:04.308538 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:04.308377    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv" podUID="3ff58008-28af-4c08-8050-5960ac71f4cf" containerName="storage-initializer" containerID="cri-o://a7b80451ba9f35d3e6b76ef7f21337291fed01b350b85ec426b3c94f015f9345" gracePeriod=30
Sep 16 18:58:05.337821 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.337800    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv_3ff58008-28af-4c08-8050-5960ac71f4cf/storage-initializer/1.log"
Sep 16 18:58:05.338289 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.338271    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv_3ff58008-28af-4c08-8050-5960ac71f4cf/storage-initializer/0.log"
Sep 16 18:58:05.338346 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.338308    2583 generic.go:356] "Generic (PLEG): container finished" podID="3ff58008-28af-4c08-8050-5960ac71f4cf" containerID="a7b80451ba9f35d3e6b76ef7f21337291fed01b350b85ec426b3c94f015f9345" exitCode=1
Sep 16 18:58:05.338382 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.338347    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv" event={"ID":"3ff58008-28af-4c08-8050-5960ac71f4cf","Type":"ContainerDied","Data":"a7b80451ba9f35d3e6b76ef7f21337291fed01b350b85ec426b3c94f015f9345"}
Sep 16 18:58:05.338440 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.338429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv" event={"ID":"3ff58008-28af-4c08-8050-5960ac71f4cf","Type":"ContainerDied","Data":"c5bdfcc5fa0e288ecd8cac3bc788b3101925c7d32cceb50e4ffe46689c3f8a7a"}
Sep 16 18:58:05.338476 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.338452    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5bdfcc5fa0e288ecd8cac3bc788b3101925c7d32cceb50e4ffe46689c3f8a7a"
Sep 16 18:58:05.338476 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.338465    2583 scope.go:117] "RemoveContainer" containerID="114d98076caf14928840f433255c9e8d709781b6b86141c00f0c84d9037433f5"
Sep 16 18:58:05.342977 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.342961    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv_3ff58008-28af-4c08-8050-5960ac71f4cf/storage-initializer/1.log"
Sep 16 18:58:05.346673 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.346656    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:58:05.495707 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.495683    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ff58008-28af-4c08-8050-5960ac71f4cf-kserve-provision-location\") pod \"3ff58008-28af-4c08-8050-5960ac71f4cf\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") "
Sep 16 18:58:05.495838 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.495734    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-cabundle-cert\") pod \"3ff58008-28af-4c08-8050-5960ac71f4cf\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") "
Sep 16 18:58:05.495838 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.495774    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-rbac-proxy-sar-config\") pod \"3ff58008-28af-4c08-8050-5960ac71f4cf\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") "
Sep 16 18:58:05.495838 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.495796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff58008-28af-4c08-8050-5960ac71f4cf-proxy-tls\") pod \"3ff58008-28af-4c08-8050-5960ac71f4cf\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") "
Sep 16 18:58:05.495838 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.495820    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f55d7\" (UniqueName: \"kubernetes.io/projected/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-api-access-f55d7\") pod \"3ff58008-28af-4c08-8050-5960ac71f4cf\" (UID: \"3ff58008-28af-4c08-8050-5960ac71f4cf\") "
Sep 16 18:58:05.496021 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.495937    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff58008-28af-4c08-8050-5960ac71f4cf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ff58008-28af-4c08-8050-5960ac71f4cf" (UID: "3ff58008-28af-4c08-8050-5960ac71f4cf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:05.496084 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.496052    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ff58008-28af-4c08-8050-5960ac71f4cf-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:58:05.496161 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.496136    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ff58008-28af-4c08-8050-5960ac71f4cf" (UID: "3ff58008-28af-4c08-8050-5960ac71f4cf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:05.496161 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.496144    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ff58008-28af-4c08-8050-5960ac71f4cf" (UID: "3ff58008-28af-4c08-8050-5960ac71f4cf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:05.497968 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.497945    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ff58008-28af-4c08-8050-5960ac71f4cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ff58008-28af-4c08-8050-5960ac71f4cf" (UID: "3ff58008-28af-4c08-8050-5960ac71f4cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:05.498089 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.498065    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-api-access-f55d7" (OuterVolumeSpecName: "kube-api-access-f55d7") pod "3ff58008-28af-4c08-8050-5960ac71f4cf" (UID: "3ff58008-28af-4c08-8050-5960ac71f4cf"). InnerVolumeSpecName "kube-api-access-f55d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:05.596689 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.596663    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:58:05.596689 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.596686    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:58:05.596845 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.596696    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ff58008-28af-4c08-8050-5960ac71f4cf-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:58:05.596845 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:05.596704    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55d7\" (UniqueName: \"kubernetes.io/projected/3ff58008-28af-4c08-8050-5960ac71f4cf-kube-api-access-f55d7\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 18:58:06.343393 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:06.343365    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv_3ff58008-28af-4c08-8050-5960ac71f4cf/storage-initializer/1.log"
Sep 16 18:58:06.343778 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:06.343454    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"
Sep 16 18:58:06.345120 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:06.345076    2583 generic.go:356] "Generic (PLEG): container finished" podID="3078bb75-1754-4187-8593-143f47243046" containerID="e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4" exitCode=0
Sep 16 18:58:06.345233 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:06.345132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" event={"ID":"3078bb75-1754-4187-8593-143f47243046","Type":"ContainerDied","Data":"e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4"}
Sep 16 18:58:06.423518 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:06.423493    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"]
Sep 16 18:58:06.428617 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:06.428595    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-ee84dd-predictor-85bd8c5df5-tcqcv"]
Sep 16 18:58:07.351347 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:07.351314    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" event={"ID":"3078bb75-1754-4187-8593-143f47243046","Type":"ContainerStarted","Data":"c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c"}
Sep 16 18:58:07.351717 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:07.351380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" event={"ID":"3078bb75-1754-4187-8593-143f47243046","Type":"ContainerStarted","Data":"0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89"}
Sep 16 18:58:07.351717 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:07.351581    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:07.372989 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:07.372941    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podStartSLOduration=2.584548998 podStartE2EDuration="6.372926954s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.788377956s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:07.370717804 +0000 UTC m=+1060.253540885" watchObservedRunningTime="2026-09-16 18:58:07.372926954 +0000 UTC m=+1060.255750034"
Sep 16 18:58:07.658046 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:07.657975    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ff58008-28af-4c08-8050-5960ac71f4cf" path="/var/lib/kubelet/pods/3ff58008-28af-4c08-8050-5960ac71f4cf/volumes"
Sep 16 18:58:08.355280 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:08.355253    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:08.356575 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:08.356549    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:58:09.366133 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:09.366096    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:58:14.370147 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:14.370119    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 18:58:14.370645 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:14.370590    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:58:24.370605 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:24.370566    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:58:34.370689 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:34.370650    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:58:44.370880 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:44.370838    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:58:54.371055 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:58:54.371012    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:59:04.371354 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:59:04.371310    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:59:14.370896 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:59:14.370855    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:59:24.371055 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:59:24.371007    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:59:34.370815 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:59:34.370771    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:59:42.654020 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:59:42.653976    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 18:59:52.654270 ip-10-0-131-137 kubenswrapper[2583]: I0916 18:59:52.654183    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:00:02.654052 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:02.654005    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:00:12.654098 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:12.654054    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:00:22.654166 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:22.654124    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:00:32.654455 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:32.654424    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 19:00:41.420168 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:41.420128    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"]
Sep 16 19:00:41.420582 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:41.420407    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" containerID="cri-o://0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89" gracePeriod=30
Sep 16 19:00:41.420582 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:41.420430    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kube-rbac-proxy" containerID="cri-o://c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c" gracePeriod=30
Sep 16 19:00:42.004400 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:42.004369    2583 generic.go:356] "Generic (PLEG): container finished" podID="3078bb75-1754-4187-8593-143f47243046" containerID="c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c" exitCode=2
Sep 16 19:00:42.004557 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:42.004442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" event={"ID":"3078bb75-1754-4187-8593-143f47243046","Type":"ContainerDied","Data":"c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c"}
Sep 16 19:00:42.654597 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:42.654554    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 16 19:00:44.366418 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:44.366374    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 16 19:00:46.719157 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.719126    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"]
Sep 16 19:00:46.719594 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.719579    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff58008-28af-4c08-8050-5960ac71f4cf" containerName="storage-initializer"
Sep 16 19:00:46.719680 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.719599    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff58008-28af-4c08-8050-5960ac71f4cf" containerName="storage-initializer"
Sep 16 19:00:46.719764 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.719699    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff58008-28af-4c08-8050-5960ac71f4cf" containerName="storage-initializer"
Sep 16 19:00:46.719830 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.719793    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff58008-28af-4c08-8050-5960ac71f4cf" containerName="storage-initializer"
Sep 16 19:00:46.719830 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.719805    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff58008-28af-4c08-8050-5960ac71f4cf" containerName="storage-initializer"
Sep 16 19:00:46.719940 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.719909    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff58008-28af-4c08-8050-5960ac71f4cf" containerName="storage-initializer"
Sep 16 19:00:46.723115 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.723096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.725199 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.725176    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:00:46.725411 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.725380    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:00:46.735208 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.735136    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"]
Sep 16 19:00:46.738930 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.738905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-497lf\" (UniqueName: \"kubernetes.io/projected/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-api-access-497lf\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.739035 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.738953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.739089 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.739042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3db31fca-c715-4627-a0e5-637a9f14c19c-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.739089 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.739080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3db31fca-c715-4627-a0e5-637a9f14c19c-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.739181 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.739105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.839788 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.839758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.839914 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.839820    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3db31fca-c715-4627-a0e5-637a9f14c19c-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.839914 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.839853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3db31fca-c715-4627-a0e5-637a9f14c19c-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.839914 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.839876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.840080 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.839917    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-497lf\" (UniqueName: \"kubernetes.io/projected/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-api-access-497lf\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.840198 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.840177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3db31fca-c715-4627-a0e5-637a9f14c19c-kserve-provision-location\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.840445 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.840421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-cabundle-cert\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.840514 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.840488    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.842428 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.842409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3db31fca-c715-4627-a0e5-637a9f14c19c-proxy-tls\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:46.853573 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:46.853544    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-497lf\" (UniqueName: \"kubernetes.io/projected/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-api-access-497lf\") pod \"isvc-canary-create-predictor-76cc455874-q8jqb\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:47.035125 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:47.035070    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:47.318037 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:47.317969    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"]
Sep 16 19:00:47.320905 ip-10-0-131-137 kubenswrapper[2583]: W0916 19:00:47.320877    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db31fca_c715_4627_a0e5_637a9f14c19c.slice/crio-1d6b755236c9a831f6c444b5a16c40eff8a5d9c41a359d1558d08e9289c93ddc WatchSource:0}: Error finding container 1d6b755236c9a831f6c444b5a16c40eff8a5d9c41a359d1558d08e9289c93ddc: Status 404 returned error can't find the container with id 1d6b755236c9a831f6c444b5a16c40eff8a5d9c41a359d1558d08e9289c93ddc
Sep 16 19:00:48.035311 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:48.035281    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" event={"ID":"3db31fca-c715-4627-a0e5-637a9f14c19c","Type":"ContainerStarted","Data":"88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b"}
Sep 16 19:00:48.035681 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:48.035337    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" event={"ID":"3db31fca-c715-4627-a0e5-637a9f14c19c","Type":"ContainerStarted","Data":"1d6b755236c9a831f6c444b5a16c40eff8a5d9c41a359d1558d08e9289c93ddc"}
Sep 16 19:00:49.367197 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:49.367140    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 16 19:00:50.875228 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:50.875204    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 19:00:50.978616 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:50.978539    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-cabundle-cert\") pod \"3078bb75-1754-4187-8593-143f47243046\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") "
Sep 16 19:00:50.978616 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:50.978579    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3078bb75-1754-4187-8593-143f47243046-kserve-provision-location\") pod \"3078bb75-1754-4187-8593-143f47243046\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") "
Sep 16 19:00:50.978844 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:50.978617    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq5j9\" (UniqueName: \"kubernetes.io/projected/3078bb75-1754-4187-8593-143f47243046-kube-api-access-vq5j9\") pod \"3078bb75-1754-4187-8593-143f47243046\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") "
Sep 16 19:00:50.978844 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:50.978649    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-kube-rbac-proxy-sar-config\") pod \"3078bb75-1754-4187-8593-143f47243046\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") "
Sep 16 19:00:50.978844 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:50.978713    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3078bb75-1754-4187-8593-143f47243046-proxy-tls\") pod \"3078bb75-1754-4187-8593-143f47243046\" (UID: \"3078bb75-1754-4187-8593-143f47243046\") "
Sep 16 19:00:50.979061 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:50.978914    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3078bb75-1754-4187-8593-143f47243046-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3078bb75-1754-4187-8593-143f47243046" (UID: "3078bb75-1754-4187-8593-143f47243046"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:50.979061 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:50.979000    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3078bb75-1754-4187-8593-143f47243046" (UID: "3078bb75-1754-4187-8593-143f47243046"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:50.979217 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:50.979077    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3078bb75-1754-4187-8593-143f47243046" (UID: "3078bb75-1754-4187-8593-143f47243046"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:50.980881 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:50.980811    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3078bb75-1754-4187-8593-143f47243046-kube-api-access-vq5j9" (OuterVolumeSpecName: "kube-api-access-vq5j9") pod "3078bb75-1754-4187-8593-143f47243046" (UID: "3078bb75-1754-4187-8593-143f47243046"). InnerVolumeSpecName "kube-api-access-vq5j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:50.981124 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:50.981050    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3078bb75-1754-4187-8593-143f47243046-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3078bb75-1754-4187-8593-143f47243046" (UID: "3078bb75-1754-4187-8593-143f47243046"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:51.050559 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.050527    2583 generic.go:356] "Generic (PLEG): container finished" podID="3078bb75-1754-4187-8593-143f47243046" containerID="0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89" exitCode=0
Sep 16 19:00:51.050689 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.050576    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"
Sep 16 19:00:51.050689 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.050602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" event={"ID":"3078bb75-1754-4187-8593-143f47243046","Type":"ContainerDied","Data":"0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89"}
Sep 16 19:00:51.050689 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.050664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq" event={"ID":"3078bb75-1754-4187-8593-143f47243046","Type":"ContainerDied","Data":"59cb3238ad7715383be0da40c410514bcff898e318bb0201ca71e50500593fc7"}
Sep 16 19:00:51.050865 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.050692    2583 scope.go:117] "RemoveContainer" containerID="c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c"
Sep 16 19:00:51.060140 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.060122    2583 scope.go:117] "RemoveContainer" containerID="0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89"
Sep 16 19:00:51.068015 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.067999    2583 scope.go:117] "RemoveContainer" containerID="e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4"
Sep 16 19:00:51.075124 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.075102    2583 scope.go:117] "RemoveContainer" containerID="c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c"
Sep 16 19:00:51.075427 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:00:51.075408    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c\": container with ID starting with c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c not found: ID does not exist" containerID="c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c"
Sep 16 19:00:51.075480 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.075437    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c"} err="failed to get container status \"c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c\": rpc error: code = NotFound desc = could not find container \"c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c\": container with ID starting with c3e55e8629577b098c04a6b2fde04556c859e2ef3ac83aedafd5deed43ac363c not found: ID does not exist"
Sep 16 19:00:51.075480 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.075454    2583 scope.go:117] "RemoveContainer" containerID="0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89"
Sep 16 19:00:51.075722 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:00:51.075708    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89\": container with ID starting with 0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89 not found: ID does not exist" containerID="0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89"
Sep 16 19:00:51.075806 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.075724    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89"} err="failed to get container status \"0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89\": rpc error: code = NotFound desc = could not find container \"0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89\": container with ID starting with 0965dcbcd007c8b574b1578b8f571c32c8293f4a0d0b2e5ef0b4900ce4f76c89 not found: ID does not exist"
Sep 16 19:00:51.075806 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.075735    2583 scope.go:117] "RemoveContainer" containerID="e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4"
Sep 16 19:00:51.076013 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:00:51.075992    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4\": container with ID starting with e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4 not found: ID does not exist" containerID="e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4"
Sep 16 19:00:51.076056 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.076024    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4"} err="failed to get container status \"e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4\": rpc error: code = NotFound desc = could not find container \"e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4\": container with ID starting with e9c90420ebe1a15b13002b11e9027deae63a630b0707f2f61e92946777bb01d4 not found: ID does not exist"
Sep 16 19:00:51.079984 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.079965    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3078bb75-1754-4187-8593-143f47243046-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:00:51.080071 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.079988    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:00:51.080071 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.080001    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3078bb75-1754-4187-8593-143f47243046-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:00:51.080071 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.080013    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq5j9\" (UniqueName: \"kubernetes.io/projected/3078bb75-1754-4187-8593-143f47243046-kube-api-access-vq5j9\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:00:51.080071 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.080026    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3078bb75-1754-4187-8593-143f47243046-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:00:51.080378 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.080359    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"]
Sep 16 19:00:51.086945 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.086925    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-cf1e0-predictor-558f6579fc-dxxjq"]
Sep 16 19:00:51.658789 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:51.658713    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3078bb75-1754-4187-8593-143f47243046" path="/var/lib/kubelet/pods/3078bb75-1754-4187-8593-143f47243046/volumes"
Sep 16 19:00:52.058294 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:52.058262    2583 generic.go:356] "Generic (PLEG): container finished" podID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerID="88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b" exitCode=0
Sep 16 19:00:52.058687 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:52.058330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" event={"ID":"3db31fca-c715-4627-a0e5-637a9f14c19c","Type":"ContainerDied","Data":"88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b"}
Sep 16 19:00:53.065426 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:53.065392    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" event={"ID":"3db31fca-c715-4627-a0e5-637a9f14c19c","Type":"ContainerStarted","Data":"4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a"}
Sep 16 19:00:53.065865 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:53.065455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" event={"ID":"3db31fca-c715-4627-a0e5-637a9f14c19c","Type":"ContainerStarted","Data":"60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a"}
Sep 16 19:00:53.065865 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:53.065727    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:53.065964 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:53.065892    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:53.067099 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:53.067075    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:00:53.088542 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:53.088493    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podStartSLOduration=3.242514002 podStartE2EDuration="7.088479453s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.845965451s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:53.08662943 +0000 UTC m=+1225.969452512" watchObservedRunningTime="2026-09-16 19:00:53.088479453 +0000 UTC m=+1225.971302533"
Sep 16 19:00:54.069935 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:54.069893    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:00:59.075006 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:59.074975    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:00:59.075556 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:00:59.075525    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:01:09.076129 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:01:09.076076    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:01:19.076240 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:01:19.076205    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:01:29.076400 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:01:29.076357    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:01:39.075408 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:01:39.075370    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:01:49.075432 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:01:49.075393    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:01:59.075672 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:01:59.075634    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 16 19:02:05.658027 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:05.657998    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:02:16.775499 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:16.775466    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"]
Sep 16 19:02:16.776131 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:16.775690    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container" containerID="cri-o://60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a" gracePeriod=30
Sep 16 19:02:16.776131 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:16.775768    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kube-rbac-proxy" containerID="cri-o://4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a" gracePeriod=30
Sep 16 19:02:17.429581 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:17.429544    2583 generic.go:356] "Generic (PLEG): container finished" podID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerID="4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a" exitCode=2
Sep 16 19:02:17.429772 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:17.429629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" event={"ID":"3db31fca-c715-4627-a0e5-637a9f14c19c","Type":"ContainerDied","Data":"4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a"}
Sep 16 19:02:19.070818 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:19.070779    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 16 19:02:22.052526 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.052498    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"]
Sep 16 19:02:22.052905 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.052823    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container"
Sep 16 19:02:22.052905 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.052835    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container"
Sep 16 19:02:22.052905 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.052849    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="storage-initializer"
Sep 16 19:02:22.052905 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.052855    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="storage-initializer"
Sep 16 19:02:22.052905 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.052864    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kube-rbac-proxy"
Sep 16 19:02:22.052905 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.052870    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kube-rbac-proxy"
Sep 16 19:02:22.053107 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.052918    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kube-rbac-proxy"
Sep 16 19:02:22.053107 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.052926    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3078bb75-1754-4187-8593-143f47243046" containerName="kserve-container"
Sep 16 19:02:22.056160 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.056144    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.059769 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.059727    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:02:22.060163 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.060144    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:02:22.069841 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.069821    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"]
Sep 16 19:02:22.188431 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.188408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b669bb-6697-4511-90b9-bbc4c330162c-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.188557 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.188440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.188557 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.188473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b669bb-6697-4511-90b9-bbc4c330162c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.188557 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.188540    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.188667 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.188585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr6w2\" (UniqueName: \"kubernetes.io/projected/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-api-access-lr6w2\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.289699 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.289674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr6w2\" (UniqueName: \"kubernetes.io/projected/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-api-access-lr6w2\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.289843 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.289726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b669bb-6697-4511-90b9-bbc4c330162c-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.289843 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.289762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.289843 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.289796    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b669bb-6697-4511-90b9-bbc4c330162c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.289843 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.289826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.290291 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.290269    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b669bb-6697-4511-90b9-bbc4c330162c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.290502 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.290482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.290558 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.290510    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-cabundle-cert\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.292343 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.292325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b669bb-6697-4511-90b9-bbc4c330162c-proxy-tls\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.300489 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.300469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr6w2\" (UniqueName: \"kubernetes.io/projected/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-api-access-lr6w2\") pod \"isvc-canary-promote-predictor-6b6f66ff94-d4tjc\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.366215 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.366163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:22.649310 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:22.649235    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"]
Sep 16 19:02:22.652420 ip-10-0-131-137 kubenswrapper[2583]: W0916 19:02:22.652391    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b669bb_6697_4511_90b9_bbc4c330162c.slice/crio-b9161731193f5dc8c5a7000281980af4ade76b3e458fcfe09a4a307a6d71ab56 WatchSource:0}: Error finding container b9161731193f5dc8c5a7000281980af4ade76b3e458fcfe09a4a307a6d71ab56: Status 404 returned error can't find the container with id b9161731193f5dc8c5a7000281980af4ade76b3e458fcfe09a4a307a6d71ab56
Sep 16 19:02:23.456590 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:23.456548    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" event={"ID":"e7b669bb-6697-4511-90b9-bbc4c330162c","Type":"ContainerStarted","Data":"6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125"}
Sep 16 19:02:23.457040 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:23.456622    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" event={"ID":"e7b669bb-6697-4511-90b9-bbc4c330162c","Type":"ContainerStarted","Data":"b9161731193f5dc8c5a7000281980af4ade76b3e458fcfe09a4a307a6d71ab56"}
Sep 16 19:02:24.126408 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.126384    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:02:24.204642 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.204614    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-cabundle-cert\") pod \"3db31fca-c715-4627-a0e5-637a9f14c19c\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") "
Sep 16 19:02:24.204792 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.204677    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3db31fca-c715-4627-a0e5-637a9f14c19c-kserve-provision-location\") pod \"3db31fca-c715-4627-a0e5-637a9f14c19c\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") "
Sep 16 19:02:24.204792 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.204756    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-rbac-proxy-sar-config\") pod \"3db31fca-c715-4627-a0e5-637a9f14c19c\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") "
Sep 16 19:02:24.204919 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.204814    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-497lf\" (UniqueName: \"kubernetes.io/projected/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-api-access-497lf\") pod \"3db31fca-c715-4627-a0e5-637a9f14c19c\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") "
Sep 16 19:02:24.204919 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.204871    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3db31fca-c715-4627-a0e5-637a9f14c19c-proxy-tls\") pod \"3db31fca-c715-4627-a0e5-637a9f14c19c\" (UID: \"3db31fca-c715-4627-a0e5-637a9f14c19c\") "
Sep 16 19:02:24.205051 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.205030    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3db31fca-c715-4627-a0e5-637a9f14c19c" (UID: "3db31fca-c715-4627-a0e5-637a9f14c19c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:24.205124 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.205027    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db31fca-c715-4627-a0e5-637a9f14c19c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3db31fca-c715-4627-a0e5-637a9f14c19c" (UID: "3db31fca-c715-4627-a0e5-637a9f14c19c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:24.205181 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.205122    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3db31fca-c715-4627-a0e5-637a9f14c19c" (UID: "3db31fca-c715-4627-a0e5-637a9f14c19c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:24.205181 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.205137    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:02:24.205262 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.205180    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3db31fca-c715-4627-a0e5-637a9f14c19c-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:02:24.206910 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.206883    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-api-access-497lf" (OuterVolumeSpecName: "kube-api-access-497lf") pod "3db31fca-c715-4627-a0e5-637a9f14c19c" (UID: "3db31fca-c715-4627-a0e5-637a9f14c19c"). InnerVolumeSpecName "kube-api-access-497lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:24.207005 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.206982    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3db31fca-c715-4627-a0e5-637a9f14c19c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3db31fca-c715-4627-a0e5-637a9f14c19c" (UID: "3db31fca-c715-4627-a0e5-637a9f14c19c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:24.306472 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.306420    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-497lf\" (UniqueName: \"kubernetes.io/projected/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-api-access-497lf\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:02:24.306472 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.306441    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3db31fca-c715-4627-a0e5-637a9f14c19c-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:02:24.306472 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.306451    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3db31fca-c715-4627-a0e5-637a9f14c19c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:02:24.462720 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.462695    2583 generic.go:356] "Generic (PLEG): container finished" podID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerID="60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a" exitCode=0
Sep 16 19:02:24.463117 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.462768    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"
Sep 16 19:02:24.463117 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.462784    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" event={"ID":"3db31fca-c715-4627-a0e5-637a9f14c19c","Type":"ContainerDied","Data":"60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a"}
Sep 16 19:02:24.463117 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.462896    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" event={"ID":"3db31fca-c715-4627-a0e5-637a9f14c19c","Type":"ContainerDied","Data":"1d6b755236c9a831f6c444b5a16c40eff8a5d9c41a359d1558d08e9289c93ddc"}
Sep 16 19:02:24.463117 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.462939    2583 scope.go:117] "RemoveContainer" containerID="4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a"
Sep 16 19:02:24.472043 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.472026    2583 scope.go:117] "RemoveContainer" containerID="60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a"
Sep 16 19:02:24.479395 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.479376    2583 scope.go:117] "RemoveContainer" containerID="88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b"
Sep 16 19:02:24.486472 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.486457    2583 scope.go:117] "RemoveContainer" containerID="4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a"
Sep 16 19:02:24.486755 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:02:24.486723    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a\": container with ID starting with 4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a not found: ID does not exist" containerID="4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a"
Sep 16 19:02:24.486812 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.486763    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a"} err="failed to get container status \"4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a\": rpc error: code = NotFound desc = could not find container \"4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a\": container with ID starting with 4a8f24e9079a9ceae0bfcba94668ad0ce43bc03af9ce08ca6d6302bca5772b0a not found: ID does not exist"
Sep 16 19:02:24.486812 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.486781    2583 scope.go:117] "RemoveContainer" containerID="60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a"
Sep 16 19:02:24.487052 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:02:24.487033    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a\": container with ID starting with 60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a not found: ID does not exist" containerID="60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a"
Sep 16 19:02:24.487123 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.487069    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a"} err="failed to get container status \"60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a\": rpc error: code = NotFound desc = could not find container \"60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a\": container with ID starting with 60a68579136160a74956306a886a6feb60978626ea69e74830e0735966d9d85a not found: ID does not exist"
Sep 16 19:02:24.487123 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.487092    2583 scope.go:117] "RemoveContainer" containerID="88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b"
Sep 16 19:02:24.487366 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:02:24.487349    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b\": container with ID starting with 88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b not found: ID does not exist" containerID="88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b"
Sep 16 19:02:24.487456 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.487368    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b"} err="failed to get container status \"88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b\": rpc error: code = NotFound desc = could not find container \"88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b\": container with ID starting with 88c543e4770705282befb664ef9fa5588af6a72bd14ce175de561f12b920011b not found: ID does not exist"
Sep 16 19:02:24.499148 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.499127    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"]
Sep 16 19:02:24.501591 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:24.501571    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb"]
Sep 16 19:02:25.071024 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:25.070984    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76cc455874-q8jqb" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": context deadline exceeded"
Sep 16 19:02:25.658611 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:25.658576    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" path="/var/lib/kubelet/pods/3db31fca-c715-4627-a0e5-637a9f14c19c/volumes"
Sep 16 19:02:27.477304 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:27.477274    2583 generic.go:356] "Generic (PLEG): container finished" podID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerID="6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125" exitCode=0
Sep 16 19:02:27.477647 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:27.477327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" event={"ID":"e7b669bb-6697-4511-90b9-bbc4c330162c","Type":"ContainerDied","Data":"6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125"}
Sep 16 19:02:28.483802 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:28.483760    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" event={"ID":"e7b669bb-6697-4511-90b9-bbc4c330162c","Type":"ContainerStarted","Data":"42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413"}
Sep 16 19:02:28.484287 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:28.483825    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" event={"ID":"e7b669bb-6697-4511-90b9-bbc4c330162c","Type":"ContainerStarted","Data":"db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d"}
Sep 16 19:02:28.484287 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:28.484090    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:28.484287 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:28.484218    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:28.485573 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:28.485544    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:02:28.506033 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:28.505990    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podStartSLOduration=2.716535403 podStartE2EDuration="6.50597391s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.789438507s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:28.505308652 +0000 UTC m=+1321.388131743" watchObservedRunningTime="2026-09-16 19:02:28.50597391 +0000 UTC m=+1321.388796991"
Sep 16 19:02:29.488206 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:29.488171    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:02:34.492949 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:34.492922    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:02:34.493461 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:34.493388    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:02:44.493472 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:44.493431    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:02:54.493497 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:02:54.493405    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:03:04.494194 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:04.494141    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:03:14.493565 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:14.493524    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:03:24.494179 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:24.494139    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:03:34.493563 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:34.493519    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 16 19:03:41.659110 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:41.659069    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:03:52.023726 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.023643    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"]
Sep 16 19:03:52.024158 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.023948    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container" containerID="cri-o://db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d" gracePeriod=30
Sep 16 19:03:52.024158 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.023993    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kube-rbac-proxy" containerID="cri-o://42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413" gracePeriod=30
Sep 16 19:03:52.078878 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.078855    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"]
Sep 16 19:03:52.079162 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.079149    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container"
Sep 16 19:03:52.079219 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.079164    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container"
Sep 16 19:03:52.079219 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.079172    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kube-rbac-proxy"
Sep 16 19:03:52.079219 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.079177    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kube-rbac-proxy"
Sep 16 19:03:52.079219 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.079194    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="storage-initializer"
Sep 16 19:03:52.079219 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.079199    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="storage-initializer"
Sep 16 19:03:52.079446 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.079252    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kserve-container"
Sep 16 19:03:52.079446 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.079259    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3db31fca-c715-4627-a0e5-637a9f14c19c" containerName="kube-rbac-proxy"
Sep 16 19:03:52.082698 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.082681    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.084634 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.084613    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:03:52.092180 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.092158    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"]
Sep 16 19:03:52.165909 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.165879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.166018 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.165923    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee158f70-62ad-4118-b69a-cc359f17325c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.166018 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.165951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vt95\" (UniqueName: \"kubernetes.io/projected/ee158f70-62ad-4118-b69a-cc359f17325c-kube-api-access-8vt95\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.166118 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.166081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.166118 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.166114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee158f70-62ad-4118-b69a-cc359f17325c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.267182 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.267155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.267274 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.267192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee158f70-62ad-4118-b69a-cc359f17325c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.267274 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.267237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.267391 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.267270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee158f70-62ad-4118-b69a-cc359f17325c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.267391 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.267296    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vt95\" (UniqueName: \"kubernetes.io/projected/ee158f70-62ad-4118-b69a-cc359f17325c-kube-api-access-8vt95\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.267655 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.267634    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee158f70-62ad-4118-b69a-cc359f17325c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.267909 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.267888    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.267990 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.267961    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.269762 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.269731    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee158f70-62ad-4118-b69a-cc359f17325c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.276877 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.276828    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vt95\" (UniqueName: \"kubernetes.io/projected/ee158f70-62ad-4118-b69a-cc359f17325c-kube-api-access-8vt95\") pod \"isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.392211 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.392177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:52.667155 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.667121    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"]
Sep 16 19:03:52.669680 ip-10-0-131-137 kubenswrapper[2583]: W0916 19:03:52.669644    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee158f70_62ad_4118_b69a_cc359f17325c.slice/crio-157f322b504edb48b64a63bbab1375decbd90a02b7cd95a82491de4305507d35 WatchSource:0}: Error finding container 157f322b504edb48b64a63bbab1375decbd90a02b7cd95a82491de4305507d35: Status 404 returned error can't find the container with id 157f322b504edb48b64a63bbab1375decbd90a02b7cd95a82491de4305507d35
Sep 16 19:03:52.842019 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.841967    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" event={"ID":"ee158f70-62ad-4118-b69a-cc359f17325c","Type":"ContainerStarted","Data":"faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3"}
Sep 16 19:03:52.842196 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.842070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" event={"ID":"ee158f70-62ad-4118-b69a-cc359f17325c","Type":"ContainerStarted","Data":"157f322b504edb48b64a63bbab1375decbd90a02b7cd95a82491de4305507d35"}
Sep 16 19:03:52.844146 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.844120    2583 generic.go:356] "Generic (PLEG): container finished" podID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerID="42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413" exitCode=2
Sep 16 19:03:52.844279 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:52.844151    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" event={"ID":"e7b669bb-6697-4511-90b9-bbc4c330162c","Type":"ContainerDied","Data":"42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413"}
Sep 16 19:03:54.488547 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:54.488506    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 16 19:03:56.069886 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.069863    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:03:56.197561 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.197537    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-cabundle-cert\") pod \"e7b669bb-6697-4511-90b9-bbc4c330162c\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") "
Sep 16 19:03:56.197699 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.197607    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b669bb-6697-4511-90b9-bbc4c330162c-kserve-provision-location\") pod \"e7b669bb-6697-4511-90b9-bbc4c330162c\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") "
Sep 16 19:03:56.197699 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.197638    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-rbac-proxy-sar-config\") pod \"e7b669bb-6697-4511-90b9-bbc4c330162c\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") "
Sep 16 19:03:56.197699 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.197669    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr6w2\" (UniqueName: \"kubernetes.io/projected/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-api-access-lr6w2\") pod \"e7b669bb-6697-4511-90b9-bbc4c330162c\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") "
Sep 16 19:03:56.197699 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.197694    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b669bb-6697-4511-90b9-bbc4c330162c-proxy-tls\") pod \"e7b669bb-6697-4511-90b9-bbc4c330162c\" (UID: \"e7b669bb-6697-4511-90b9-bbc4c330162c\") "
Sep 16 19:03:56.197985 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.197958    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b669bb-6697-4511-90b9-bbc4c330162c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7b669bb-6697-4511-90b9-bbc4c330162c" (UID: "e7b669bb-6697-4511-90b9-bbc4c330162c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:56.198037 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.197984    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7b669bb-6697-4511-90b9-bbc4c330162c" (UID: "e7b669bb-6697-4511-90b9-bbc4c330162c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:56.198140 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.198118    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7b669bb-6697-4511-90b9-bbc4c330162c" (UID: "e7b669bb-6697-4511-90b9-bbc4c330162c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:56.199927 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.199897    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b669bb-6697-4511-90b9-bbc4c330162c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7b669bb-6697-4511-90b9-bbc4c330162c" (UID: "e7b669bb-6697-4511-90b9-bbc4c330162c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:56.200018 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.199985    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-api-access-lr6w2" (OuterVolumeSpecName: "kube-api-access-lr6w2") pod "e7b669bb-6697-4511-90b9-bbc4c330162c" (UID: "e7b669bb-6697-4511-90b9-bbc4c330162c"). InnerVolumeSpecName "kube-api-access-lr6w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:56.299170 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.299142    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7b669bb-6697-4511-90b9-bbc4c330162c-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:03:56.299170 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.299165    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:03:56.299170 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.299175    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr6w2\" (UniqueName: \"kubernetes.io/projected/e7b669bb-6697-4511-90b9-bbc4c330162c-kube-api-access-lr6w2\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:03:56.299359 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.299184    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7b669bb-6697-4511-90b9-bbc4c330162c-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:03:56.299359 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.299194    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7b669bb-6697-4511-90b9-bbc4c330162c-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:03:56.864655 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.864622    2583 generic.go:356] "Generic (PLEG): container finished" podID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerID="db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d" exitCode=0
Sep 16 19:03:56.864831 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.864689    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"
Sep 16 19:03:56.864831 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.864713    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" event={"ID":"e7b669bb-6697-4511-90b9-bbc4c330162c","Type":"ContainerDied","Data":"db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d"}
Sep 16 19:03:56.864831 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.864816    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc" event={"ID":"e7b669bb-6697-4511-90b9-bbc4c330162c","Type":"ContainerDied","Data":"b9161731193f5dc8c5a7000281980af4ade76b3e458fcfe09a4a307a6d71ab56"}
Sep 16 19:03:56.864997 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.864851    2583 scope.go:117] "RemoveContainer" containerID="42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413"
Sep 16 19:03:56.866775 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.866731    2583 generic.go:356] "Generic (PLEG): container finished" podID="ee158f70-62ad-4118-b69a-cc359f17325c" containerID="faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3" exitCode=0
Sep 16 19:03:56.866917 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.866788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" event={"ID":"ee158f70-62ad-4118-b69a-cc359f17325c","Type":"ContainerDied","Data":"faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3"}
Sep 16 19:03:56.875008 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.874986    2583 scope.go:117] "RemoveContainer" containerID="db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d"
Sep 16 19:03:56.883497 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.883474    2583 scope.go:117] "RemoveContainer" containerID="6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125"
Sep 16 19:03:56.896568 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.896550    2583 scope.go:117] "RemoveContainer" containerID="42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413"
Sep 16 19:03:56.897375 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:03:56.897351    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413\": container with ID starting with 42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413 not found: ID does not exist" containerID="42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413"
Sep 16 19:03:56.897479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.897382    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413"} err="failed to get container status \"42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413\": rpc error: code = NotFound desc = could not find container \"42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413\": container with ID starting with 42c6e6562227d53b4fa982386a6c9c5c0ce0ccf71fdc849eea4918cff6d1d413 not found: ID does not exist"
Sep 16 19:03:56.897479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.897404    2583 scope.go:117] "RemoveContainer" containerID="db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d"
Sep 16 19:03:56.897772 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:03:56.897729    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d\": container with ID starting with db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d not found: ID does not exist" containerID="db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d"
Sep 16 19:03:56.897839 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.897781    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d"} err="failed to get container status \"db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d\": rpc error: code = NotFound desc = could not find container \"db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d\": container with ID starting with db3f2033548103d557a3bccf6a7382dca5df59758190ba780cacf68b19e9b69d not found: ID does not exist"
Sep 16 19:03:56.897839 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.897796    2583 scope.go:117] "RemoveContainer" containerID="6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125"
Sep 16 19:03:56.898125 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:03:56.898104    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125\": container with ID starting with 6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125 not found: ID does not exist" containerID="6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125"
Sep 16 19:03:56.898184 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.898131    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125"} err="failed to get container status \"6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125\": rpc error: code = NotFound desc = could not find container \"6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125\": container with ID starting with 6484f70328c10fd3d0e0a1c9fa4fabee0383f801070247c579647de7f3728125 not found: ID does not exist"
Sep 16 19:03:56.915313 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.915290    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"]
Sep 16 19:03:56.925394 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:56.925373    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6b6f66ff94-d4tjc"]
Sep 16 19:03:57.659187 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:57.659155    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" path="/var/lib/kubelet/pods/e7b669bb-6697-4511-90b9-bbc4c330162c/volumes"
Sep 16 19:03:57.874369 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:57.874344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" event={"ID":"ee158f70-62ad-4118-b69a-cc359f17325c","Type":"ContainerStarted","Data":"5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183"}
Sep 16 19:03:57.874511 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:57.874395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" event={"ID":"ee158f70-62ad-4118-b69a-cc359f17325c","Type":"ContainerStarted","Data":"65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e"}
Sep 16 19:03:57.874570 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:57.874549    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:57.894500 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:57.894451    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podStartSLOduration=2.319777254 podStartE2EDuration="5.894432907s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.574655653s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:57.892225387 +0000 UTC m=+1410.775048468" watchObservedRunningTime="2026-09-16 19:03:57.894432907 +0000 UTC m=+1410.777255988"
Sep 16 19:03:58.878850 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:58.878815    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:03:58.880091 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:58.880066    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:03:59.883873 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:03:59.883821    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:04:04.888496 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:04:04.888466    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:04:04.888935 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:04:04.888911    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:04:14.889624 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:04:14.889576    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:04:24.889434 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:04:24.889395    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:04:27.713271 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:04:27.713239    2583 scope.go:117] "RemoveContainer" containerID="a7b80451ba9f35d3e6b76ef7f21337291fed01b350b85ec426b3c94f015f9345"
Sep 16 19:04:34.889488 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:04:34.889453    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:04:44.889002 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:04:44.888956    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:04:54.889962 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:04:54.889924    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:05:04.889402 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:04.889359    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:05:14.889892 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:14.889861    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:05:22.275077 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:22.275043    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"]
Sep 16 19:05:22.275539 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:22.275314    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container" containerID="cri-o://65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e" gracePeriod=30
Sep 16 19:05:22.275539 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:22.275340    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kube-rbac-proxy" containerID="cri-o://5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183" gracePeriod=30
Sep 16 19:05:23.238294 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:23.238254    2583 generic.go:356] "Generic (PLEG): container finished" podID="ee158f70-62ad-4118-b69a-cc359f17325c" containerID="5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183" exitCode=2
Sep 16 19:05:23.238461 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:23.238326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" event={"ID":"ee158f70-62ad-4118-b69a-cc359f17325c","Type":"ContainerDied","Data":"5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183"}
Sep 16 19:05:24.884318 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:24.884281    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 16 19:05:24.889615 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:24.889583    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 16 19:05:26.614360 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.614336    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:05:26.728665 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.728636    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee158f70-62ad-4118-b69a-cc359f17325c-proxy-tls\") pod \"ee158f70-62ad-4118-b69a-cc359f17325c\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") "
Sep 16 19:05:26.728830 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.728673    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee158f70-62ad-4118-b69a-cc359f17325c-kserve-provision-location\") pod \"ee158f70-62ad-4118-b69a-cc359f17325c\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") "
Sep 16 19:05:26.728830 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.728719    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vt95\" (UniqueName: \"kubernetes.io/projected/ee158f70-62ad-4118-b69a-cc359f17325c-kube-api-access-8vt95\") pod \"ee158f70-62ad-4118-b69a-cc359f17325c\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") "
Sep 16 19:05:26.728830 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.728800    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-kube-rbac-proxy-sar-config\") pod \"ee158f70-62ad-4118-b69a-cc359f17325c\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") "
Sep 16 19:05:26.728830 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.728824    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-cabundle-cert\") pod \"ee158f70-62ad-4118-b69a-cc359f17325c\" (UID: \"ee158f70-62ad-4118-b69a-cc359f17325c\") "
Sep 16 19:05:26.729092 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.729070    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee158f70-62ad-4118-b69a-cc359f17325c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee158f70-62ad-4118-b69a-cc359f17325c" (UID: "ee158f70-62ad-4118-b69a-cc359f17325c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:26.729247 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.729212    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee158f70-62ad-4118-b69a-cc359f17325c" (UID: "ee158f70-62ad-4118-b69a-cc359f17325c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:26.729344 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.729255    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ee158f70-62ad-4118-b69a-cc359f17325c" (UID: "ee158f70-62ad-4118-b69a-cc359f17325c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:26.730914 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.730886    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee158f70-62ad-4118-b69a-cc359f17325c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee158f70-62ad-4118-b69a-cc359f17325c" (UID: "ee158f70-62ad-4118-b69a-cc359f17325c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:26.731001 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.730973    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee158f70-62ad-4118-b69a-cc359f17325c-kube-api-access-8vt95" (OuterVolumeSpecName: "kube-api-access-8vt95") pod "ee158f70-62ad-4118-b69a-cc359f17325c" (UID: "ee158f70-62ad-4118-b69a-cc359f17325c"). InnerVolumeSpecName "kube-api-access-8vt95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:26.829978 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.829945    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:05:26.829978 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.829976    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ee158f70-62ad-4118-b69a-cc359f17325c-cabundle-cert\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:05:26.829978 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.829985    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee158f70-62ad-4118-b69a-cc359f17325c-proxy-tls\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:05:26.830143 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.829993    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee158f70-62ad-4118-b69a-cc359f17325c-kserve-provision-location\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:05:26.830143 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:26.830001    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vt95\" (UniqueName: \"kubernetes.io/projected/ee158f70-62ad-4118-b69a-cc359f17325c-kube-api-access-8vt95\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:05:27.257391 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.257357    2583 generic.go:356] "Generic (PLEG): container finished" podID="ee158f70-62ad-4118-b69a-cc359f17325c" containerID="65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e" exitCode=0
Sep 16 19:05:27.257551 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.257413    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"
Sep 16 19:05:27.257551 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.257439    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" event={"ID":"ee158f70-62ad-4118-b69a-cc359f17325c","Type":"ContainerDied","Data":"65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e"}
Sep 16 19:05:27.257551 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.257498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl" event={"ID":"ee158f70-62ad-4118-b69a-cc359f17325c","Type":"ContainerDied","Data":"157f322b504edb48b64a63bbab1375decbd90a02b7cd95a82491de4305507d35"}
Sep 16 19:05:27.257551 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.257525    2583 scope.go:117] "RemoveContainer" containerID="5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183"
Sep 16 19:05:27.266252 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.266235    2583 scope.go:117] "RemoveContainer" containerID="65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e"
Sep 16 19:05:27.273675 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.273657    2583 scope.go:117] "RemoveContainer" containerID="faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3"
Sep 16 19:05:27.280854 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.280833    2583 scope.go:117] "RemoveContainer" containerID="5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183"
Sep 16 19:05:27.281158 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:05:27.281138    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183\": container with ID starting with 5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183 not found: ID does not exist" containerID="5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183"
Sep 16 19:05:27.281265 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.281169    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183"} err="failed to get container status \"5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183\": rpc error: code = NotFound desc = could not find container \"5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183\": container with ID starting with 5be7c6edece89814e16110796eb263a20954b57a73b63e2cb0781e0c33dd9183 not found: ID does not exist"
Sep 16 19:05:27.281265 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.281190    2583 scope.go:117] "RemoveContainer" containerID="65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e"
Sep 16 19:05:27.281453 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:05:27.281437    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e\": container with ID starting with 65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e not found: ID does not exist" containerID="65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e"
Sep 16 19:05:27.281494 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.281460    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e"} err="failed to get container status \"65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e\": rpc error: code = NotFound desc = could not find container \"65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e\": container with ID starting with 65310f6b1ce0db7d406387c7dfc7d2fb1654cd2459dd5a6f1c998bac8859858e not found: ID does not exist"
Sep 16 19:05:27.281494 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.281475    2583 scope.go:117] "RemoveContainer" containerID="faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3"
Sep 16 19:05:27.281817 ip-10-0-131-137 kubenswrapper[2583]: E0916 19:05:27.281792    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3\": container with ID starting with faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3 not found: ID does not exist" containerID="faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3"
Sep 16 19:05:27.281905 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.281823    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3"} err="failed to get container status \"faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3\": rpc error: code = NotFound desc = could not find container \"faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3\": container with ID starting with faecc0fc4c2267bc3fac1153c63f31fad22c91474f32fd25aea5a8ca11609ec3 not found: ID does not exist"
Sep 16 19:05:27.282182 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.282164    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"]
Sep 16 19:05:27.289292 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.289274    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-769cdc7dcc-6b4bl"]
Sep 16 19:05:27.659246 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:05:27.659149    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" path="/var/lib/kubelet/pods/ee158f70-62ad-4118-b69a-cc359f17325c/volumes"
Sep 16 19:07:23.294055 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294019    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vbfj9/must-gather-s77bn"]
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294330    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294341    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294351    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kube-rbac-proxy"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294356    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kube-rbac-proxy"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294364    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kube-rbac-proxy"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294370    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kube-rbac-proxy"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294384    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294389    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294395    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="storage-initializer"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294400    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="storage-initializer"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294406    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="storage-initializer"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294411    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="storage-initializer"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294460    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kube-rbac-proxy"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294469    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee158f70-62ad-4118-b69a-cc359f17325c" containerName="kserve-container"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294476    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kserve-container"
Sep 16 19:07:23.296479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.294481    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b669bb-6697-4511-90b9-bbc4c330162c" containerName="kube-rbac-proxy"
Sep 16 19:07:23.297330 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.297314    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbfj9/must-gather-s77bn"
Sep 16 19:07:23.299758 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.299725    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vbfj9\"/\"default-dockercfg-shpwv\""
Sep 16 19:07:23.299894 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.299728    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbfj9\"/\"kube-root-ca.crt\""
Sep 16 19:07:23.299894 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.299846    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbfj9\"/\"openshift-service-ca.crt\""
Sep 16 19:07:23.313205 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.313183    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbfj9/must-gather-s77bn"]
Sep 16 19:07:23.421650 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.421613    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-must-gather-output\") pod \"must-gather-s77bn\" (UID: \"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4\") " pod="openshift-must-gather-vbfj9/must-gather-s77bn"
Sep 16 19:07:23.421857 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.421680    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbxrl\" (UniqueName: \"kubernetes.io/projected/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-kube-api-access-rbxrl\") pod \"must-gather-s77bn\" (UID: \"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4\") " pod="openshift-must-gather-vbfj9/must-gather-s77bn"
Sep 16 19:07:23.522423 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.522385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbxrl\" (UniqueName: \"kubernetes.io/projected/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-kube-api-access-rbxrl\") pod \"must-gather-s77bn\" (UID: \"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4\") " pod="openshift-must-gather-vbfj9/must-gather-s77bn"
Sep 16 19:07:23.522597 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.522449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-must-gather-output\") pod \"must-gather-s77bn\" (UID: \"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4\") " pod="openshift-must-gather-vbfj9/must-gather-s77bn"
Sep 16 19:07:23.522790 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.522772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-must-gather-output\") pod \"must-gather-s77bn\" (UID: \"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4\") " pod="openshift-must-gather-vbfj9/must-gather-s77bn"
Sep 16 19:07:23.533411 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.533381    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbxrl\" (UniqueName: \"kubernetes.io/projected/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-kube-api-access-rbxrl\") pod \"must-gather-s77bn\" (UID: \"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4\") " pod="openshift-must-gather-vbfj9/must-gather-s77bn"
Sep 16 19:07:23.606478 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.606391    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbfj9/must-gather-s77bn"
Sep 16 19:07:23.885093 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.885060    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbfj9/must-gather-s77bn"]
Sep 16 19:07:23.887532 ip-10-0-131-137 kubenswrapper[2583]: W0916 19:07:23.887503    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69ea1e2a_4889_4c3f_b7cb_32d4dc22d2a4.slice/crio-48c3a911dbac2dd748b2bda1c8c83915d87678206e3e0826e9c12ce5e529441c WatchSource:0}: Error finding container 48c3a911dbac2dd748b2bda1c8c83915d87678206e3e0826e9c12ce5e529441c: Status 404 returned error can't find the container with id 48c3a911dbac2dd748b2bda1c8c83915d87678206e3e0826e9c12ce5e529441c
Sep 16 19:07:23.889489 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:23.889466    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:07:24.728043 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:24.727977    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbfj9/must-gather-s77bn" event={"ID":"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4","Type":"ContainerStarted","Data":"48c3a911dbac2dd748b2bda1c8c83915d87678206e3e0826e9c12ce5e529441c"}
Sep 16 19:07:30.759658 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:30.759614    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbfj9/must-gather-s77bn" event={"ID":"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4","Type":"ContainerStarted","Data":"4b592e29983663e9a8943d774e6cada1d7689108994b55b9fee875c36741c12d"}
Sep 16 19:07:30.760095 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:30.759669    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbfj9/must-gather-s77bn" event={"ID":"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4","Type":"ContainerStarted","Data":"0fdb564771cf92ae33f30920b1f0069640797d484fb30c0da8b0bff5c05f99ac"}
Sep 16 19:07:30.778883 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:30.778838    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vbfj9/must-gather-s77bn" podStartSLOduration=1.741074986 podStartE2EDuration="7.778822685s" totalImagesPullingTime="6.037747699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:30.777232342 +0000 UTC m=+1623.660055423" watchObservedRunningTime="2026-09-16 19:07:30.778822685 +0000 UTC m=+1623.661645800"
Sep 16 19:07:53.865905 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:53.865867    2583 generic.go:356] "Generic (PLEG): container finished" podID="69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" containerID="0fdb564771cf92ae33f30920b1f0069640797d484fb30c0da8b0bff5c05f99ac" exitCode=0
Sep 16 19:07:53.866347 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:53.865939    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbfj9/must-gather-s77bn" event={"ID":"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4","Type":"ContainerDied","Data":"0fdb564771cf92ae33f30920b1f0069640797d484fb30c0da8b0bff5c05f99ac"}
Sep 16 19:07:53.866347 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:53.866253    2583 scope.go:117] "RemoveContainer" containerID="0fdb564771cf92ae33f30920b1f0069640797d484fb30c0da8b0bff5c05f99ac"
Sep 16 19:07:54.070866 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.070831    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vbfj9_must-gather-s77bn_69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4/gather/0.log"
Sep 16 19:07:54.740490 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.740448    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9gvsp/must-gather-mpztb"]
Sep 16 19:07:54.743878 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.743854    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gvsp/must-gather-mpztb"
Sep 16 19:07:54.746660 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.746639    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9gvsp\"/\"openshift-service-ca.crt\""
Sep 16 19:07:54.746660 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.746650    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9gvsp\"/\"default-dockercfg-lwr2v\""
Sep 16 19:07:54.746848 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.746644    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9gvsp\"/\"kube-root-ca.crt\""
Sep 16 19:07:54.758322 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.758301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gvsp/must-gather-mpztb"]
Sep 16 19:07:54.809775 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.809731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhzsx\" (UniqueName: \"kubernetes.io/projected/0ea7ba9d-7f9a-4114-ae00-da991237e63e-kube-api-access-xhzsx\") pod \"must-gather-mpztb\" (UID: \"0ea7ba9d-7f9a-4114-ae00-da991237e63e\") " pod="openshift-must-gather-9gvsp/must-gather-mpztb"
Sep 16 19:07:54.809938 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.809873    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0ea7ba9d-7f9a-4114-ae00-da991237e63e-must-gather-output\") pod \"must-gather-mpztb\" (UID: \"0ea7ba9d-7f9a-4114-ae00-da991237e63e\") " pod="openshift-must-gather-9gvsp/must-gather-mpztb"
Sep 16 19:07:54.910617 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.910585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0ea7ba9d-7f9a-4114-ae00-da991237e63e-must-gather-output\") pod \"must-gather-mpztb\" (UID: \"0ea7ba9d-7f9a-4114-ae00-da991237e63e\") " pod="openshift-must-gather-9gvsp/must-gather-mpztb"
Sep 16 19:07:54.911005 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.910625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhzsx\" (UniqueName: \"kubernetes.io/projected/0ea7ba9d-7f9a-4114-ae00-da991237e63e-kube-api-access-xhzsx\") pod \"must-gather-mpztb\" (UID: \"0ea7ba9d-7f9a-4114-ae00-da991237e63e\") " pod="openshift-must-gather-9gvsp/must-gather-mpztb"
Sep 16 19:07:54.911005 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.910913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0ea7ba9d-7f9a-4114-ae00-da991237e63e-must-gather-output\") pod \"must-gather-mpztb\" (UID: \"0ea7ba9d-7f9a-4114-ae00-da991237e63e\") " pod="openshift-must-gather-9gvsp/must-gather-mpztb"
Sep 16 19:07:54.921100 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:54.921074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhzsx\" (UniqueName: \"kubernetes.io/projected/0ea7ba9d-7f9a-4114-ae00-da991237e63e-kube-api-access-xhzsx\") pod \"must-gather-mpztb\" (UID: \"0ea7ba9d-7f9a-4114-ae00-da991237e63e\") " pod="openshift-must-gather-9gvsp/must-gather-mpztb"
Sep 16 19:07:55.052703 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:55.052628    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gvsp/must-gather-mpztb"
Sep 16 19:07:55.328990 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:55.328915    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gvsp/must-gather-mpztb"]
Sep 16 19:07:55.331981 ip-10-0-131-137 kubenswrapper[2583]: W0916 19:07:55.331952    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ea7ba9d_7f9a_4114_ae00_da991237e63e.slice/crio-b71129155a94471cc2e7bf910e22d46027a2097c96460c918ab7f6af43a29ad8 WatchSource:0}: Error finding container b71129155a94471cc2e7bf910e22d46027a2097c96460c918ab7f6af43a29ad8: Status 404 returned error can't find the container with id b71129155a94471cc2e7bf910e22d46027a2097c96460c918ab7f6af43a29ad8
Sep 16 19:07:55.874880 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:55.874843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gvsp/must-gather-mpztb" event={"ID":"0ea7ba9d-7f9a-4114-ae00-da991237e63e","Type":"ContainerStarted","Data":"b71129155a94471cc2e7bf910e22d46027a2097c96460c918ab7f6af43a29ad8"}
Sep 16 19:07:56.881634 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:56.881597    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gvsp/must-gather-mpztb" event={"ID":"0ea7ba9d-7f9a-4114-ae00-da991237e63e","Type":"ContainerStarted","Data":"aed7e9df84740d04623181bde0958bcef65ba8bb92e2f03d09579d55d8e3deb6"}
Sep 16 19:07:56.882123 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:56.881649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gvsp/must-gather-mpztb" event={"ID":"0ea7ba9d-7f9a-4114-ae00-da991237e63e","Type":"ContainerStarted","Data":"4e267a2d24f89daaf6bab3aab03abe58be4f830c0cea73f034de62434484806d"}
Sep 16 19:07:56.901064 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:56.900993    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9gvsp/must-gather-mpztb" podStartSLOduration=2.335474789 podStartE2EDuration="2.900972327s" totalImagesPullingTime="565.497538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:56.900186447 +0000 UTC m=+1649.783009529" watchObservedRunningTime="2026-09-16 19:07:56.900972327 +0000 UTC m=+1649.783795409"
Sep 16 19:07:57.508871 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:57.508844    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7gt7h_2b45a65e-e81b-45fe-8f11-db4a8685bc46/global-pull-secret-syncer/0.log"
Sep 16 19:07:57.721666 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:57.721629    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m5v5w_a30106ce-a8b7-4549-b466-343652ad0736/konnectivity-agent/0.log"
Sep 16 19:07:57.777936 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:57.777834    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-137.ec2.internal_ae92e6bf8b06378903825aae1b8252ca/haproxy/0.log"
Sep 16 19:07:59.537079 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:59.537029    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-vbfj9/must-gather-s77bn"]
Sep 16 19:07:59.537673 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:59.537566    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-vbfj9/must-gather-s77bn" podUID="69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" containerName="copy" containerID="cri-o://4b592e29983663e9a8943d774e6cada1d7689108994b55b9fee875c36741c12d" gracePeriod=2
Sep 16 19:07:59.543347 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:59.543319    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-vbfj9/must-gather-s77bn"]
Sep 16 19:07:59.911793 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:59.906337    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vbfj9_must-gather-s77bn_69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4/copy/0.log"
Sep 16 19:07:59.911793 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:59.908151    2583 generic.go:356] "Generic (PLEG): container finished" podID="69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" containerID="4b592e29983663e9a8943d774e6cada1d7689108994b55b9fee875c36741c12d" exitCode=143
Sep 16 19:07:59.931497 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:59.931452    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vbfj9_must-gather-s77bn_69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4/copy/0.log"
Sep 16 19:07:59.938107 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:07:59.937985    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbfj9/must-gather-s77bn"
Sep 16 19:08:00.071553 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:00.071510    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-must-gather-output\") pod \"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4\" (UID: \"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4\") "
Sep 16 19:08:00.071785 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:00.071607    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbxrl\" (UniqueName: \"kubernetes.io/projected/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-kube-api-access-rbxrl\") pod \"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4\" (UID: \"69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4\") "
Sep 16 19:08:00.074265 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:00.074230    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" (UID: "69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:00.075701 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:00.075673    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-kube-api-access-rbxrl" (OuterVolumeSpecName: "kube-api-access-rbxrl") pod "69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" (UID: "69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4"). InnerVolumeSpecName "kube-api-access-rbxrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:00.173655 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:00.173551    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-must-gather-output\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:08:00.173655 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:00.173608    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbxrl\" (UniqueName: \"kubernetes.io/projected/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4-kube-api-access-rbxrl\") on node \"ip-10-0-131-137.ec2.internal\" DevicePath \"\""
Sep 16 19:08:00.916199 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:00.916164    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vbfj9_must-gather-s77bn_69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4/copy/0.log"
Sep 16 19:08:00.916803 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:00.916763    2583 scope.go:117] "RemoveContainer" containerID="4b592e29983663e9a8943d774e6cada1d7689108994b55b9fee875c36741c12d"
Sep 16 19:08:00.916958 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:00.916939    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbfj9/must-gather-s77bn"
Sep 16 19:08:00.932838 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:00.932708    2583 scope.go:117] "RemoveContainer" containerID="0fdb564771cf92ae33f30920b1f0069640797d484fb30c0da8b0bff5c05f99ac"
Sep 16 19:08:01.661854 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:01.661811    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" path="/var/lib/kubelet/pods/69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4/volumes"
Sep 16 19:08:02.586590 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:02.586549    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-wlrhq_31c13446-60a1-4b20-a197-cb3f08fcf388/kube-state-metrics/0.log"
Sep 16 19:08:02.621836 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:02.621720    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-wlrhq_31c13446-60a1-4b20-a197-cb3f08fcf388/kube-rbac-proxy-main/0.log"
Sep 16 19:08:02.652107 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:02.652080    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-wlrhq_31c13446-60a1-4b20-a197-cb3f08fcf388/kube-rbac-proxy-self/0.log"
Sep 16 19:08:02.700589 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:02.700542    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-64bf7496bf-767ls_63654fe1-e483-4ce6-accb-662e776408e6/metrics-server/0.log"
Sep 16 19:08:02.738802 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:02.738771    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-cw54s_d2e60300-e117-4a73-831f-6be65e3ac1c5/monitoring-plugin/0.log"
Sep 16 19:08:02.900601 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:02.900525    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t62ml_c7a1769b-c075-40ca-a814-6b09b1b7ac62/node-exporter/0.log"
Sep 16 19:08:02.936074 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:02.936039    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t62ml_c7a1769b-c075-40ca-a814-6b09b1b7ac62/kube-rbac-proxy/0.log"
Sep 16 19:08:02.967715 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:02.967678    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t62ml_c7a1769b-c075-40ca-a814-6b09b1b7ac62/init-textfile/0.log"
Sep 16 19:08:03.311774 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.310767    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82e57f8-4c77-4a9a-bea8-85f1b1394d3b/prometheus/0.log"
Sep 16 19:08:03.347989 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.345832    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82e57f8-4c77-4a9a-bea8-85f1b1394d3b/config-reloader/0.log"
Sep 16 19:08:03.387920 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.387890    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82e57f8-4c77-4a9a-bea8-85f1b1394d3b/thanos-sidecar/0.log"
Sep 16 19:08:03.420465 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.420401    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82e57f8-4c77-4a9a-bea8-85f1b1394d3b/kube-rbac-proxy-web/0.log"
Sep 16 19:08:03.454697 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.454461    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82e57f8-4c77-4a9a-bea8-85f1b1394d3b/kube-rbac-proxy/0.log"
Sep 16 19:08:03.491479 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.491447    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82e57f8-4c77-4a9a-bea8-85f1b1394d3b/kube-rbac-proxy-thanos/0.log"
Sep 16 19:08:03.527131 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.527055    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82e57f8-4c77-4a9a-bea8-85f1b1394d3b/init-config-reloader/0.log"
Sep 16 19:08:03.577067 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.577028    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-pv6hl_3da8c784-e626-469b-8d9e-b9c0e7d9e44a/prometheus-operator/0.log"
Sep 16 19:08:03.615058 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.615027    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-pv6hl_3da8c784-e626-469b-8d9e-b9c0e7d9e44a/kube-rbac-proxy/0.log"
Sep 16 19:08:03.647766 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.647710    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-qv69d_04989fa3-1e2a-40cc-9624-3c061a9e88a5/prometheus-operator-admission-webhook/0.log"
Sep 16 19:08:03.694069 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.694036    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f9d8c4559-xzdtc_e07a579f-0139-4090-a990-0537655d9ba5/telemeter-client/0.log"
Sep 16 19:08:03.726006 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.725977    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f9d8c4559-xzdtc_e07a579f-0139-4090-a990-0537655d9ba5/reload/0.log"
Sep 16 19:08:03.756768 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:03.756719    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f9d8c4559-xzdtc_e07a579f-0139-4090-a990-0537655d9ba5/kube-rbac-proxy/0.log"
Sep 16 19:08:05.878457 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:05.878421    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"]
Sep 16 19:08:05.878940 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:05.878732    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" containerName="copy"
Sep 16 19:08:05.878940 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:05.878763    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" containerName="copy"
Sep 16 19:08:05.878940 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:05.878790    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" containerName="gather"
Sep 16 19:08:05.878940 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:05.878796    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" containerName="gather"
Sep 16 19:08:05.878940 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:05.878864    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" containerName="gather"
Sep 16 19:08:05.878940 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:05.878874    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ea1e2a-4889-4c3f-b7cb-32d4dc22d2a4" containerName="copy"
Sep 16 19:08:05.882428 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:05.882403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:05.892140 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:05.892111    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"]
Sep 16 19:08:06.039608 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.039577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-podres\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.039811 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.039634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-proc\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.039811 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.039691    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-lib-modules\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.039811 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.039760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-sys\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.039811 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.039805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txf6\" (UniqueName: \"kubernetes.io/projected/f38ecbdc-4419-4ac7-9938-8733daa37f47-kube-api-access-4txf6\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.044314 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.044293    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74f6bf7ddb-l7lld_fe2097f6-50b1-4311-843c-108cbec24d7f/console/0.log"
Sep 16 19:08:06.141216 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.141107    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-podres\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.141216 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.141202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-proc\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.141442 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.141245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-lib-modules\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.141442 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.141286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-sys\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.141442 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.141287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-podres\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.141442 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.141340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txf6\" (UniqueName: \"kubernetes.io/projected/f38ecbdc-4419-4ac7-9938-8733daa37f47-kube-api-access-4txf6\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.141442 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.141359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-sys\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.141442 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.141380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-lib-modules\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.141788 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.141465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f38ecbdc-4419-4ac7-9938-8733daa37f47-proc\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.153568 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.153534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txf6\" (UniqueName: \"kubernetes.io/projected/f38ecbdc-4419-4ac7-9938-8733daa37f47-kube-api-access-4txf6\") pod \"perf-node-gather-daemonset-w9xg4\" (UID: \"f38ecbdc-4419-4ac7-9938-8733daa37f47\") " pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.198665 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.198622    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.484368 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.484292    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"]
Sep 16 19:08:06.489735 ip-10-0-131-137 kubenswrapper[2583]: W0916 19:08:06.489689    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf38ecbdc_4419_4ac7_9938_8733daa37f47.slice/crio-f2af01b13f1b34ac13c63d81ebecb66c2267d3930714271c0e7a51d4ace1fa0f WatchSource:0}: Error finding container f2af01b13f1b34ac13c63d81ebecb66c2267d3930714271c0e7a51d4ace1fa0f: Status 404 returned error can't find the container with id f2af01b13f1b34ac13c63d81ebecb66c2267d3930714271c0e7a51d4ace1fa0f
Sep 16 19:08:06.950552 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.950516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4" event={"ID":"f38ecbdc-4419-4ac7-9938-8733daa37f47","Type":"ContainerStarted","Data":"e5e7d646a88231c1e06062117eb81af70a1012c363c51ca44053b813c705e868"}
Sep 16 19:08:06.951022 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.950583    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4" event={"ID":"f38ecbdc-4419-4ac7-9938-8733daa37f47","Type":"ContainerStarted","Data":"f2af01b13f1b34ac13c63d81ebecb66c2267d3930714271c0e7a51d4ace1fa0f"}
Sep 16 19:08:06.951022 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.950645    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:06.971961 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:06.971888    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4" podStartSLOduration=1.9718654199999999 podStartE2EDuration="1.97186542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:06.970157695 +0000 UTC m=+1659.852980793" watchObservedRunningTime="2026-09-16 19:08:06.97186542 +0000 UTC m=+1659.854688502"
Sep 16 19:08:07.729126 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:07.729073    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2qbqd_aeb3b2b9-dea0-4346-9418-9b9394b13d82/dns/0.log"
Sep 16 19:08:07.759058 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:07.759029    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2qbqd_aeb3b2b9-dea0-4346-9418-9b9394b13d82/kube-rbac-proxy/0.log"
Sep 16 19:08:07.882730 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:07.882702    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gt9pt_ae2b5270-597d-4a80-8f88-908422bec8a9/dns-node-resolver/0.log"
Sep 16 19:08:08.426822 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:08.426791    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9tntl_c3e6df42-134d-4588-bb47-5163ecb7925b/node-ca/0.log"
Sep 16 19:08:09.701514 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:09.701480    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5nt49_f9947aa6-6269-453c-9a91-e95f47fffac3/serve-healthcheck-canary/0.log"
Sep 16 19:08:10.182880 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:10.182847    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-jcr5s_7761e9ff-e826-4316-9a1a-876df7e23311/insights-operator/0.log"
Sep 16 19:08:10.183603 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:10.183580    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-jcr5s_7761e9ff-e826-4316-9a1a-876df7e23311/insights-operator/1.log"
Sep 16 19:08:10.296434 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:10.296413    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-njs2v_293464f7-a7bc-4529-8a46-a3266ec67b0a/kube-rbac-proxy/0.log"
Sep 16 19:08:10.327209 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:10.327186    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-njs2v_293464f7-a7bc-4529-8a46-a3266ec67b0a/exporter/0.log"
Sep 16 19:08:10.359452 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:10.359427    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-njs2v_293464f7-a7bc-4529-8a46-a3266ec67b0a/extractor/0.log"
Sep 16 19:08:12.965573 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:12.965548    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9gvsp/perf-node-gather-daemonset-w9xg4"
Sep 16 19:08:13.255902 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:13.255823    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-s9fqh_3c1f70ee-4ca7-4550-b098-bcd450197a28/s3-init/0.log"
Sep 16 19:08:13.284941 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:13.284923    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-s9fqh_3c1f70ee-4ca7-4550-b098-bcd450197a28/download-hf-model/0.log"
Sep 16 19:08:19.724425 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:19.724391    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrtmh_38dd8c04-24c6-4a26-bdae-c9cb85d20453/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:08:19.751754 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:19.751709    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrtmh_38dd8c04-24c6-4a26-bdae-c9cb85d20453/egress-router-binary-copy/0.log"
Sep 16 19:08:19.779850 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:19.779820    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrtmh_38dd8c04-24c6-4a26-bdae-c9cb85d20453/cni-plugins/0.log"
Sep 16 19:08:19.805534 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:19.805511    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrtmh_38dd8c04-24c6-4a26-bdae-c9cb85d20453/bond-cni-plugin/0.log"
Sep 16 19:08:19.834189 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:19.834165    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrtmh_38dd8c04-24c6-4a26-bdae-c9cb85d20453/routeoverride-cni/0.log"
Sep 16 19:08:19.866086 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:19.866059    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrtmh_38dd8c04-24c6-4a26-bdae-c9cb85d20453/whereabouts-cni-bincopy/0.log"
Sep 16 19:08:19.898149 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:19.898122    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nrtmh_38dd8c04-24c6-4a26-bdae-c9cb85d20453/whereabouts-cni/0.log"
Sep 16 19:08:20.181858 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:20.181820    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nt66b_87639fb2-3cf6-4d7c-b488-f00772b8f76b/kube-multus/0.log"
Sep 16 19:08:20.306462 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:20.306433    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n9vzt_13bef125-8a64-4554-9ef9-fb4ddabc0628/network-metrics-daemon/0.log"
Sep 16 19:08:20.332627 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:20.332597    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n9vzt_13bef125-8a64-4554-9ef9-fb4ddabc0628/kube-rbac-proxy/0.log"
Sep 16 19:08:21.492859 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:21.492824    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n9rcp_351b8c67-9d1b-4e1f-9e3d-f108f1f39058/ovn-controller/0.log"
Sep 16 19:08:21.516989 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:21.516965    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n9rcp_351b8c67-9d1b-4e1f-9e3d-f108f1f39058/ovn-acl-logging/0.log"
Sep 16 19:08:21.545585 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:21.545555    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n9rcp_351b8c67-9d1b-4e1f-9e3d-f108f1f39058/kube-rbac-proxy-node/0.log"
Sep 16 19:08:21.571584 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:21.571556    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n9rcp_351b8c67-9d1b-4e1f-9e3d-f108f1f39058/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:08:21.595668 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:21.595643    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n9rcp_351b8c67-9d1b-4e1f-9e3d-f108f1f39058/northd/0.log"
Sep 16 19:08:21.622196 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:21.622169    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n9rcp_351b8c67-9d1b-4e1f-9e3d-f108f1f39058/nbdb/0.log"
Sep 16 19:08:21.649323 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:21.649294    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n9rcp_351b8c67-9d1b-4e1f-9e3d-f108f1f39058/sbdb/0.log"
Sep 16 19:08:21.704526 ip-10-0-131-137 kubenswrapper[2583]: I0916 19:08:21.704494    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n9rcp_351b8c67-9d1b-4e1f-9e3d-f108f1f39058/ovnkube-controller/0.log"