Sep 17 18:27:12.789141 ip-10-0-130-229 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 18:27:12.789319 ip-10-0-130-229 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 18:27:12.789424 ip-10-0-130-229 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 18:27:12.790058 ip-10-0-130-229 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 18:27:23.038073 ip-10-0-130-229 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 18:27:23.038089 ip-10-0-130-229 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 470dda2f856d495dbd2d43dd6511ce93 --
Sep 17 18:29:24.424233 ip-10-0-130-229 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:29:28.033757 ip-10-0-130-229 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:28.033757 ip-10-0-130-229 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:29:28.033757 ip-10-0-130-229 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:28.033757 ip-10-0-130-229 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:29:28.033757 ip-10-0-130-229 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:29:28.034473 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.034385    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:29:28.038183 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038167    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:28.038183 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038182    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038187    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038193    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038196    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038200    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038204    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038206    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038210    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038212    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038215    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038218    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038221    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038223    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038228    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038231    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038234    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038237    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038239    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038242    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:28.038246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038251    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038254    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038257    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038260    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038263    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038266    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038268    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038271    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038273    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038276    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038278    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038281    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038283    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038286    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038289    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038292    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038294    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038297    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038299    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038302    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:28.038687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038304    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038307    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038309    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038312    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038314    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038316    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038319    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038321    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038324    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038326    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038328    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038331    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038333    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038335    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038338    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038341    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038344    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038347    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038349    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038351    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:28.039188 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038354    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038356    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038361    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038364    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038366    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038369    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038372    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038375    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038378    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038381    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038383    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038386    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038389    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038392    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038395    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038397    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038399    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038402    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038404    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038407    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:28.039656 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038409    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038412    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038414    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038417    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038419    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038422    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038425    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038428    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038430    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038433    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038435    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038437    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038440    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038442    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038445    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038447    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038450    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038452    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.038455    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:28.040148 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039184    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039192    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039195    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039198    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039201    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039204    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039206    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039210    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039213    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039216    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039218    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039220    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039223    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039226    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039228    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039231    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039233    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039236    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039239    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039242    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:28.040689 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039245    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039248    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039251    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039253    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039256    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039259    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039261    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039264    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039266    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039268    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039271    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039273    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039276    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039280    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039282    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039285    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039287    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039290    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039292    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:28.041226 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039297    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039301    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039304    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039307    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039310    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039312    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039315    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039317    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039320    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039323    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039325    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039329    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039333    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039335    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039338    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039341    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039343    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039346    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039348    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039351    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:28.041692 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039353    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039356    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039358    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039361    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039363    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039365    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039368    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039371    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039375    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039378    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039381    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039383    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039385    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039388    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039390    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039393    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039396    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039399    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039401    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039404    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:28.042245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039406    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039409    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039411    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039414    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039416    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039419    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039422    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039424    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039427    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039430    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039433    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039437    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039440    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039443    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039445    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039448    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039451    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039454    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039457    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039460    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:28.042733 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039836    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039841    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039843    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039846    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039849    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039852    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039855    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039857    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039860    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039863    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039865    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039868    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039871    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039875    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039894    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039897    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039900    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039904    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:28.043232 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039907    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039910    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039913    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039916    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039918    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039922    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039925    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039928    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039931    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039934    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039937    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039939    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039942    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039945    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039947    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039950    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039953    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039955    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039958    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039961    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:28.043700 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039964    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039967    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039969    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039973    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039975    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039977    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039980    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039982    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039985    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039988    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039990    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039993    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039996    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.039998    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040001    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040004    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040006    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040009    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040013    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040016    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:28.044246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040018    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040021    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040023    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040026    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040029    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040031    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040034    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040036    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040039    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040043    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040047    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040050    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040053    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040055    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040058    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040061    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040063    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040066    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040068    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:28.044728 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040070    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040073    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040075    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040079    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040082    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040084    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040087    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040090    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040093    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040096    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040099    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040103    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040105    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040109    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040111    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040113    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040116    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040118    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040121    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040123    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:28.045251 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040126    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040128    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040212    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040253    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040262    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040266    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040271    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040274    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040279    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040284    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040287    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040290    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040293    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040297    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040299    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040303    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040306    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040308    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040312    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040314    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040319    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040322    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040325    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040328    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:29:28.045731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040332    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040336    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040339    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040342    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040345    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040348    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040351    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040353    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040356    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040359    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040363    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040366    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040368    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040371    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040374    2582 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040377    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040381    2582 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040384    2582 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040387    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040390    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040393    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040396    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040399    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040402    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040405    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:29:28.046347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040407    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040410    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040413    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040416    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040419    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040422    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040425    2582 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040429    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040432    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040435    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040438    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040441    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040444    2582 flags.go:64] FLAG: --help="false"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040447    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040450    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040453    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040456    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040459    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040463    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040465    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040468    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040471    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040474    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:29:28.047046 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040476    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040479    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040483    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040486    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040488    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040491    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040494    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040497    2582 flags.go:64] FLAG: --lock-file=""
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040500    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040503    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040506    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040511    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040514    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040516    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040519    2582 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040522    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040525    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040528    2582 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040531    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040535    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040538    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040542    2582 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040544    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040547    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040550    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:29:28.047656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040553    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040555    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040558    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040561    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040569    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040572    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040575    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040578    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040581    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040587    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040591    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040594    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040598    2582 flags.go:64] FLAG: --port="10250"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040601    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040603    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a0e7b3f4a8fb0183"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040606    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040610    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040613    2582 flags.go:64] FLAG: --register-node="true"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040615    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040619    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040622    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040625    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040628    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040632    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040635    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:29:28.048358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040638    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040640    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040643    2582 flags.go:64] FLAG: --runonce="false"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040646    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040649    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040652    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040654    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040657    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040660    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040663    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040666    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040669    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040672    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040675    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040678    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040681    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040684    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040686    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040692    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040697    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040700    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040704    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040708    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040711    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040714    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:29:28.048994 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040717    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040720    2582 flags.go:64] FLAG: --v="2"
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040726    2582 flags.go:64] FLAG: --version="false"
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040733    2582 flags.go:64] FLAG: --vmodule=""
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040739    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.040744    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040863    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040867    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040870    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040873    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040876    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040891    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040894    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040898    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040900    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040903    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040906    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040908    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040911    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040913    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040916    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:28.049610 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040919    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040921    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040931    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040934    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040937    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040942    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040944    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040947    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040951    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040954    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040956    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040959    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040962    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040966    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040969    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040973    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040977    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040980    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040983    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040985    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:28.050139 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040988    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040991    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040993    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040996    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.040999    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041001    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041004    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041006    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041009    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041012    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041015    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041017    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041019    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041022    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041024    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041027    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041029    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041033    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041037    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041041    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:28.050616 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041046    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041051    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041055    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041058    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041060    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041063    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041066    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041069    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041072    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041074    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041076    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041079    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041081    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041084    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041086    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041089    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041091    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041094    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041096    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:28.051114 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041099    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041101    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041104    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041107    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041109    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041112    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041114    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041117    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041119    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041122    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041126    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041128    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041131    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041135    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041139    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041142    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041145    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041148    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041150    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:28.051602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041153    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041156    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041158    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041161    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041164    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.041167    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.041172    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.048563    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.048579    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.048600    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.048606    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048669    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048675    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048678    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048681    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:28.052262 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048685    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048688    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048691    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048693    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048696    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048698    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048701    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048703    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048706    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048709    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048713    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048716    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048719    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048722    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048725    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048728    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048730    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048733    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048736    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:28.052663 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048739    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048743    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048746    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048749    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048751    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048754    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048758    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048761    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048763    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048766    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048770    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048772    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048775    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048777    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048780    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048783    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048786    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048788    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048791    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:28.053245 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048793    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048796    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048798    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048800    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048803    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048806    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048809    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048811    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048814    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048816    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048819    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048821    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048823    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048826    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048828    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048831    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048833    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048835    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048838    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048841    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048844    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:28.053842 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048847    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048849    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048852    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048855    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048857    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048860    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048862    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048865    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048867    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048870    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048872    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048875    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048877    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048896    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048899    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048901    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048903    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048907    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048909    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:28.054407 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048912    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048916    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048919    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048932    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048936    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048939    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048942    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048946    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048950    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048953    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048955    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048959    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048962    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048965    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048967    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048970    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:28.054865 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.048973    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.048978    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049088    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049094    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049097    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049100    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049103    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049105    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049108    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049111    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049131    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049135    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049138    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049140    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049144    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:29:28.055478 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049147    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049150    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049152    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049155    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049158    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049160    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049163    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049165    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049168    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049171    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049173    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049176    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049178    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049181    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049184    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049187    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049189    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049192    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049195    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049197    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:29:28.056175 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049200    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049202    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049205    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049207    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049211    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049214    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049217    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049220    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049223    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049226    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049228    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049231    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049233    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049236    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049239    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049241    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049244    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049248    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049251    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:29:28.056951 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049254    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049256    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049259    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049261    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049264    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049266    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049269    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049272    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049274    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049277    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049279    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049282    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049284    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049287    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049289    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049292    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049294    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049297    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049299    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049302    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:29:28.057442 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049304    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049307    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049309    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049312    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049314    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049316    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049319    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049322    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049324    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049327    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049329    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049332    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049334    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049336    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049339    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049342    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049346    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049348    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049354    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:29:28.057976 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049357    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049360    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049363    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049366    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049369    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049371    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049374    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:28.049377    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.049381    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.049493    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.051688    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.052576    2582 server.go:1019] "Starting client certificate rotation"
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.052691    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:28.058499 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.052733    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:29:28.058828 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.055730    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:28.058828 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.058207    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:29:28.063555 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.063535    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:29:28.065375 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.065357    2582 log.go:25] "Validated CRI v1 image API"
Sep 17 18:29:28.066220 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.066205    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:29:28.068719 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.068696    2582 fs.go:135] Filesystem UUIDs: map[4240210b-a613-4d9e-9ade-f020293154a6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e6a66d0b-fef4-406a-bccb-f992c3d36d58:/dev/nvme0n1p3]
Sep 17 18:29:28.068772 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.068720    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:29:28.073830 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.073714    2582 manager.go:216] Machine: {Timestamp:2026-09-17 18:29:28.072384872 +0000 UTC m=+1.996167524 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115676 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ebb40f746360a22d827d9a6381701 SystemUUID:ec2ebb40-f746-360a-22d8-27d9a6381701 BootID:470dda2f-856d-495d-bd2d-43dd6511ce93 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:21:dc:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:21:dc:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:8d:7c:12:85:64 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:29:28.073830 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.073830    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:29:28.073980 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.073966    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:29:28.075597 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.075573    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:29:28.075750 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.075600    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-229.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:29:28.076999 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.076988    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:29:28.077039 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.077002    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:29:28.077039 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.077010    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:28.077039 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.077020    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:29:28.077118 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.077095    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:29:28.081360 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.081345    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:28.081492 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.081482    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:29:28.081526 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.081521    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:29:28.081567 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.081534    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:29:28.081567 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.081562    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:29:28.081626 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.081577    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:29:28.081626 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.081590    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:29:28.082121 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.082103    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:28.082199 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.082129    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:29:28.084526 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.084509    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:29:28.087519 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.087498    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:29:28.087604 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.087554    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:29:28.088893 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.088866    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:29:28.088967 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.088903    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:29:28.088967 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.088915    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:29:28.088967 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.088921    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:29:28.088967 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.088945    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:29:28.088967 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.088953    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:29:28.088967 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.088966    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:29:28.089132 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.088971    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:29:28.089132 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.088978    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:29:28.089132 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.088985    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:29:28.089132 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.089000    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:29:28.089132 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.089005    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:29:28.089132 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.089039    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:29:28.089132 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.089048    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:29:28.093339 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.093313    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:28.093845 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.093825    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-229.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:29:28.094463 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.094441    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:29:28.094568 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.094445    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:29:28.094568 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.094503    2582 server.go:1268] "Started kubelet"
Sep 17 18:29:28.094658 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.094600    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:29:28.094658 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.094618    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:29:28.094747 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.094679    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:29:28.095449 ip-10-0-130-229 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:29:28.096028 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.096013    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:29:28.097317 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.097302    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:29:28.100370 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.100344    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:29:28.103308 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.103294    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:29:28.103459 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.103309    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:28.103459 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.103379    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:28.103910 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.103895    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:29:28.104090 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.103936    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:29:28.104090 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.104088    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:29:28.104232 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.104222    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:29:28.104232 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.104232    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:29:28.104484 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.104470    2582 factory.go:153] Registering CRI-O factory
Sep 17 18:29:28.104553 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.104490    2582 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:29:28.104553 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.104551    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:29:28.104649 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.104560    2582 factory.go:55] Registering systemd factory
Sep 17 18:29:28.104649 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.104569    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:29:28.104649 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.104587    2582 factory.go:103] Registering Raw factory
Sep 17 18:29:28.104649 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.104605    2582 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:29:28.105101 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.105088    2582 manager.go:349] Starting recovery of all containers
Sep 17 18:29:28.107016 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.106986    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:29:28.107127 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.107103    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:29:28.114849 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.114828    2582 manager.go:354] Recovery completed
Sep 17 18:29:28.116322 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.116299    2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 17 18:29:28.128724 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.128695    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4kxnd"
Sep 17 18:29:28.128724 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.128722    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4kxnd"
Sep 17 18:29:28.131785 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.131763    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:28.134574 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.134556    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:28.134643 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.134589    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:28.134643 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.134604    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:28.135165 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.135150    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:29:28.135165 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.135165    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:29:28.135297 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.135182    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:29:28.135297 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.135224    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:28.136376 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.136363    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:28.136429 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.136390    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:28.136429 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.136402    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:28.137214 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.137201    2582 policy_none.go:49] "None policy: Start"
Sep 17 18:29:28.137269 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.137221    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:29:28.137269 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.137236    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:29:28.139173 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.139160    2582 policy_none.go:47] "Start"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.174679    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.176002    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.176046    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.176068    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.176120    2582 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.178639    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.183542    2582 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.183589    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.183602    2582 server.go:87] "Starting device plugin registration server"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.183901    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.183921    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.184008    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.184079    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.184087    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.184597    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:29:28.187863 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.184643    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:28.277274 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.277228    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-229.ec2.internal"]
Sep 17 18:29:28.277432 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.277323    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:28.278418 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.278402    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:28.278486 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.278435    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:28.278486 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.278448    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:28.280813 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.280802    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:28.281016 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.280995    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.281054 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.281035    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:28.281660 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.281638    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:28.281779 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.281670    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:28.281779 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.281686    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:28.281779 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.281685    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:28.281949 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.281813    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:28.281949 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.281834    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:28.284012 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.283949    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.284012 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.283976    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:28.284124 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.284010    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:29:28.284782 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.284766    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:28.284860 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.284794    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:28.284860 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.284809    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:28.284860 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.284814    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:29:28.284860 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.284831    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:29:28.284860 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.284843    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:29:28.285063 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.284877    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.293546 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.293528    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.293631 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.293549    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-229.ec2.internal\": node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:28.305656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.305626    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ece2644ee37debecdc1aee17a00ea8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal\" (UID: \"9ece2644ee37debecdc1aee17a00ea8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.305768 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.305661    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ece2644ee37debecdc1aee17a00ea8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal\" (UID: \"9ece2644ee37debecdc1aee17a00ea8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.305768 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.305683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41cc877ab68985cd82a0459700bd3eff-config\") pod \"kube-apiserver-proxy-ip-10-0-130-229.ec2.internal\" (UID: \"41cc877ab68985cd82a0459700bd3eff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.309262 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.309244    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:28.314654 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.314636    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-229.ec2.internal\" not found" node="ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.319109 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.319093    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-229.ec2.internal\" not found" node="ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.406070 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.406039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ece2644ee37debecdc1aee17a00ea8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal\" (UID: \"9ece2644ee37debecdc1aee17a00ea8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.406070 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.406067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ece2644ee37debecdc1aee17a00ea8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal\" (UID: \"9ece2644ee37debecdc1aee17a00ea8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.406263 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.406086    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41cc877ab68985cd82a0459700bd3eff-config\") pod \"kube-apiserver-proxy-ip-10-0-130-229.ec2.internal\" (UID: \"41cc877ab68985cd82a0459700bd3eff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.406263 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.406131    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/41cc877ab68985cd82a0459700bd3eff-config\") pod \"kube-apiserver-proxy-ip-10-0-130-229.ec2.internal\" (UID: \"41cc877ab68985cd82a0459700bd3eff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.406263 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.406154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ece2644ee37debecdc1aee17a00ea8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal\" (UID: \"9ece2644ee37debecdc1aee17a00ea8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.406263 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.406175    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ece2644ee37debecdc1aee17a00ea8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal\" (UID: \"9ece2644ee37debecdc1aee17a00ea8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.410146 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.410117    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:28.511094 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.511061    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:28.611828 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.611797    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:28.618985 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.618955    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.623411 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:28.623389    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:28.712358 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.712322    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:28.812868 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.812836    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:28.913396 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:28.913309    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:29.013839 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:29.013804    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:29.045176 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.045142    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:29.052648 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.052622    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:29:29.052767 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.052743    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:29.052808 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.052771    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:29:29.114390 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:29.114360    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-229.ec2.internal\" not found"
Sep 17 18:29:29.135499 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.135455    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 18:24:28 +0000 UTC" deadline="2028-04-25 00:26:28.441434301 +0000 UTC"
Sep 17 18:29:29.135499 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.135495    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14045h56m59.305943491s"
Sep 17 18:29:29.165727 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.165512    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:29.204060 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.204027    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:29.215998 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.215973    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:29.216830 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.216812    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.ec2.internal"
Sep 17 18:29:29.225649 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.225626    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:29:29.281331 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:29.281291    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cc877ab68985cd82a0459700bd3eff.slice/crio-c528735f2fe0b603f1715c380ca6c9ccd2013034f1c1156fd18d42aad76bf5ab WatchSource:0}: Error finding container c528735f2fe0b603f1715c380ca6c9ccd2013034f1c1156fd18d42aad76bf5ab: Status 404 returned error can't find the container with id c528735f2fe0b603f1715c380ca6c9ccd2013034f1c1156fd18d42aad76bf5ab
Sep 17 18:29:29.281783 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:29.281751    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ece2644ee37debecdc1aee17a00ea8d.slice/crio-ab5b215816ab63cfae315b847b9ec3d1b96acdee7e9eed7dbe11ae0f5f752ffe WatchSource:0}: Error finding container ab5b215816ab63cfae315b847b9ec3d1b96acdee7e9eed7dbe11ae0f5f752ffe: Status 404 returned error can't find the container with id ab5b215816ab63cfae315b847b9ec3d1b96acdee7e9eed7dbe11ae0f5f752ffe
Sep 17 18:29:29.286595 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.286571    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:29:29.582285 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:29.582249    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:30.082802 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.082767    2582 apiserver.go:52] "Watching apiserver"
Sep 17 18:29:30.088483 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.088458    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:29:30.088834 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.088803    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-229.ec2.internal","openshift-cluster-node-tuning-operator/tuned-65s5c","openshift-dns/node-resolver-4brlg","openshift-image-registry/node-ca-p75rj","openshift-multus/multus-additional-cni-plugins-5259j","openshift-multus/network-metrics-daemon-rsp6q","openshift-network-operator/iptables-alerter-8tph9","kube-system/global-pull-secret-syncer-5v9qj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal","openshift-multus/multus-smrws","openshift-network-diagnostics/network-check-target-dt469","openshift-ovn-kubernetes/ovnkube-node-p9vs8","kube-system/konnectivity-agent-rwlgb"]
Sep 17 18:29:30.091385 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.091362    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8tph9"
Sep 17 18:29:30.094036 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.093618    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:30.094036 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.093914    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:29:30.094036 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.094015    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:30.094036 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.094027    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s6thp\""
Sep 17 18:29:30.094272 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.094140    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.096031 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.096004    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4brlg"
Sep 17 18:29:30.096400 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.096008    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:29:30.096499 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.096485    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:29:30.096554 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.096529    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:29:30.096603 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.096571    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:29:30.096682 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.096666    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tmvt\""
Sep 17 18:29:30.096715 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.096707    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:29:30.096749 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.096729    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:29:30.098274 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.098252    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ss8rl\""
Sep 17 18:29:30.098546 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.098527    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:29:30.098546 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.098544    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:29:30.101176 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.098907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p75rj"
Sep 17 18:29:30.101176 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.100914    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:29:30.101176 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.100923    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bwktv\""
Sep 17 18:29:30.101176 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.100928    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:29:30.101176 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.101013    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:29:30.101471 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.101286    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.105294 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.103584    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:29:30.105294 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.103603    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:29:30.105294 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.103796    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:29:30.105294 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.103806    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g766x\""
Sep 17 18:29:30.105294 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.103931    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:29:30.105294 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.104066    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:29:30.105294 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.104217    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:29:30.106262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.105899    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:30.106262 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.105988    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:30.108417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.108397    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.112982 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.110735    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:29:30.112982 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.110777    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pmtgv\""
Sep 17 18:29:30.112982 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.110780    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:29:30.112982 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.111094    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:30.112982 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.111628    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:29:30.114503 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.114450    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:29:30.114610 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.114574    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9shpq\""
Sep 17 18:29:30.115001 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.114942    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.115524 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.115504    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:29:30.116478 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.116437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.116650 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.116625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-kubelet\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117164 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-slash\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117258 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117217    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-ovnkube-script-lib\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117314 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76a82b7-6e16-4354-92aa-da9e6b01b704-tmp-dir\") pod \"node-resolver-4brlg\" (UID: \"a76a82b7-6e16-4354-92aa-da9e6b01b704\") " pod="openshift-dns/node-resolver-4brlg"
Sep 17 18:29:30.117314 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-systemd-units\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117454 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117344    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-run-openvswitch\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117454 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmvnq\" (UniqueName: \"kubernetes.io/projected/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-kube-api-access-cmvnq\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117454 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6wn\" (UniqueName: \"kubernetes.io/projected/3ef8e676-9989-483a-be1f-751f64eddfe7-kube-api-access-gh6wn\") pod \"node-ca-p75rj\" (UID: \"3ef8e676-9989-483a-be1f-751f64eddfe7\") " pod="openshift-image-registry/node-ca-p75rj"
Sep 17 18:29:30.117548 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117489    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8528fdf6-c48e-4f4d-bdc4-a685de05b47d-iptables-alerter-script\") pod \"iptables-alerter-8tph9\" (UID: \"8528fdf6-c48e-4f4d-bdc4-a685de05b47d\") " pod="openshift-network-operator/iptables-alerter-8tph9"
Sep 17 18:29:30.117548 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-node-log\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117548 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-log-socket\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117654 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-env-overrides\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117654 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-system-cni-dir\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.117654 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117609    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.117654 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117766 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-ovn-node-metrics-cert\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117766 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117696    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-run-netns\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117766 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117722    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-ovnkube-config\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117766 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hz88\" (UniqueName: \"kubernetes.io/projected/a76a82b7-6e16-4354-92aa-da9e6b01b704-kube-api-access-8hz88\") pod \"node-resolver-4brlg\" (UID: \"a76a82b7-6e16-4354-92aa-da9e6b01b704\") " pod="openshift-dns/node-resolver-4brlg"
Sep 17 18:29:30.117903 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117772    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.117903 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-var-lib-openvswitch\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117903 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-etc-openvswitch\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.117903 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-cni-bin\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.118031 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117915    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a76a82b7-6e16-4354-92aa-da9e6b01b704-hosts-file\") pod \"node-resolver-4brlg\" (UID: \"a76a82b7-6e16-4354-92aa-da9e6b01b704\") " pod="openshift-dns/node-resolver-4brlg"
Sep 17 18:29:30.118031 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ef8e676-9989-483a-be1f-751f64eddfe7-host\") pod \"node-ca-p75rj\" (UID: \"3ef8e676-9989-483a-be1f-751f64eddfe7\") " pod="openshift-image-registry/node-ca-p75rj"
Sep 17 18:29:30.118031 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.117959    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ef8e676-9989-483a-be1f-751f64eddfe7-serviceca\") pod \"node-ca-p75rj\" (UID: \"3ef8e676-9989-483a-be1f-751f64eddfe7\") " pod="openshift-image-registry/node-ca-p75rj"
Sep 17 18:29:30.118031 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmc9d\" (UniqueName: \"kubernetes.io/projected/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-kube-api-access-hmc9d\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.118167 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vjmm\" (UniqueName: \"kubernetes.io/projected/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-kube-api-access-6vjmm\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:30.118167 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118052    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:29:30.118167 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118141    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:29:30.118167 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118056    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wqmb\" (UniqueName: \"kubernetes.io/projected/8528fdf6-c48e-4f4d-bdc4-a685de05b47d-kube-api-access-7wqmb\") pod \"iptables-alerter-8tph9\" (UID: \"8528fdf6-c48e-4f4d-bdc4-a685de05b47d\") " pod="openshift-network-operator/iptables-alerter-8tph9"
Sep 17 18:29:30.118279 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-run-ovn\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.118279 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.118279 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118231    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-cnibin\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.118279 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118252    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.118403 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:30.118403 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8528fdf6-c48e-4f4d-bdc4-a685de05b47d-host-slash\") pod \"iptables-alerter-8tph9\" (UID: \"8528fdf6-c48e-4f4d-bdc4-a685de05b47d\") " pod="openshift-network-operator/iptables-alerter-8tph9"
Sep 17 18:29:30.118403 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-run-systemd\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.118403 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-cni-netd\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.118556 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118494    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:30.118707 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.118561    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:30.118707 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.118640    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v6hp4\""
Sep 17 18:29:30.120502 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.120481    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:29:30.121986 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.121967    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.124021 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.124002    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:29:30.124120 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.124021    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qnzd9\""
Sep 17 18:29:30.125033 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.125013    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:30.125125 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.125089    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:30.142136 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.142113    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sswht"
Sep 17 18:29:30.149507 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.149484    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sswht"
Sep 17 18:29:30.181598 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.181502    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal" event={"ID":"9ece2644ee37debecdc1aee17a00ea8d","Type":"ContainerStarted","Data":"ab5b215816ab63cfae315b847b9ec3d1b96acdee7e9eed7dbe11ae0f5f752ffe"}
Sep 17 18:29:30.183284 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.183255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.ec2.internal" event={"ID":"41cc877ab68985cd82a0459700bd3eff","Type":"ContainerStarted","Data":"c528735f2fe0b603f1715c380ca6c9ccd2013034f1c1156fd18d42aad76bf5ab"}
Sep 17 18:29:30.205128 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.205055    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:29:30.218802 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.218765    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-var-lib-cni-bin\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.218802 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.218803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/676e15c7-92c5-4d69-b2d4-7cc98fe94e86-konnectivity-ca\") pod \"konnectivity-agent-rwlgb\" (UID: \"676e15c7-92c5-4d69-b2d4-7cc98fe94e86\") " pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:30.219055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.218831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-sysconfig\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.219055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.218862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-run-netns\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.219055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.218907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.219055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.218926    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-dbus\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:30.219055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.218947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-var-lib-cni-multus\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.219055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.218971    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-hostroot\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.219055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.218989    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-run-netns\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.219055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.218993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-lib-modules\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.219055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-tuned\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.219055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-var-lib-openvswitch\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-etc-openvswitch\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219095    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vjmm\" (UniqueName: \"kubernetes.io/projected/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-kube-api-access-6vjmm\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wqmb\" (UniqueName: \"kubernetes.io/projected/8528fdf6-c48e-4f4d-bdc4-a685de05b47d-kube-api-access-7wqmb\") pod \"iptables-alerter-8tph9\" (UID: \"8528fdf6-c48e-4f4d-bdc4-a685de05b47d\") " pod="openshift-network-operator/iptables-alerter-8tph9"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219132    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-etc-openvswitch\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-cnibin\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-var-lib-openvswitch\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-daemon-config\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-systemd\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-run-systemd\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219373    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-run-systemd\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-cnibin\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.219591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-socket-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zskpw\" (UniqueName: \"kubernetes.io/projected/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-kube-api-access-zskpw\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-kubelet\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-slash\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-ovnkube-script-lib\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-var-lib-kubelet\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-kubelet\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219706    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-cnibin\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-slash\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76a82b7-6e16-4354-92aa-da9e6b01b704-tmp-dir\") pod \"node-resolver-4brlg\" (UID: \"a76a82b7-6e16-4354-92aa-da9e6b01b704\") " pod="openshift-dns/node-resolver-4brlg"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.219996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmvnq\" (UniqueName: \"kubernetes.io/projected/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-kube-api-access-cmvnq\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6wn\" (UniqueName: \"kubernetes.io/projected/3ef8e676-9989-483a-be1f-751f64eddfe7-kube-api-access-gh6wn\") pod \"node-ca-p75rj\" (UID: \"3ef8e676-9989-483a-be1f-751f64eddfe7\") " pod="openshift-image-registry/node-ca-p75rj"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220071    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-device-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220134    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-system-cni-dir\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220168    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76a82b7-6e16-4354-92aa-da9e6b01b704-tmp-dir\") pod \"node-resolver-4brlg\" (UID: \"a76a82b7-6e16-4354-92aa-da9e6b01b704\") " pod="openshift-dns/node-resolver-4brlg"
Sep 17 18:29:30.220417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-run-k8s-cni-cncf-io\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220243    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-env-overrides\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-system-cni-dir\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220321    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-kubelet-config\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/676e15c7-92c5-4d69-b2d4-7cc98fe94e86-agent-certs\") pod \"konnectivity-agent-rwlgb\" (UID: \"676e15c7-92c5-4d69-b2d4-7cc98fe94e86\") " pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-system-cni-dir\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220391    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-kubernetes\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220434    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h244l\" (UniqueName: \"kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l\") pod \"network-check-target-dt469\" (UID: \"24e1408d-b93a-449f-bd8e-6dfb76828d48\") " pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220453    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-registration-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-sysctl-d\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-host\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220521    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-ovnkube-config\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hz88\" (UniqueName: \"kubernetes.io/projected/a76a82b7-6e16-4354-92aa-da9e6b01b704-kube-api-access-8hz88\") pod \"node-resolver-4brlg\" (UID: \"a76a82b7-6e16-4354-92aa-da9e6b01b704\") " pod="openshift-dns/node-resolver-4brlg"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220560    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-cni-dir\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.221135 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-etc-kubernetes\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-sysctl-conf\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220629    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-env-overrides\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-cni-bin\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a76a82b7-6e16-4354-92aa-da9e6b01b704-hosts-file\") pod \"node-resolver-4brlg\" (UID: \"a76a82b7-6e16-4354-92aa-da9e6b01b704\") " pod="openshift-dns/node-resolver-4brlg"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220685    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-ovnkube-script-lib\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220721    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ef8e676-9989-483a-be1f-751f64eddfe7-host\") pod \"node-ca-p75rj\" (UID: \"3ef8e676-9989-483a-be1f-751f64eddfe7\") " pod="openshift-image-registry/node-ca-p75rj"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220714    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-cni-bin\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220737    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a76a82b7-6e16-4354-92aa-da9e6b01b704-hosts-file\") pod \"node-resolver-4brlg\" (UID: \"a76a82b7-6e16-4354-92aa-da9e6b01b704\") " pod="openshift-dns/node-resolver-4brlg"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ef8e676-9989-483a-be1f-751f64eddfe7-serviceca\") pod \"node-ca-p75rj\" (UID: \"3ef8e676-9989-483a-be1f-751f64eddfe7\") " pod="openshift-image-registry/node-ca-p75rj"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.220774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ef8e676-9989-483a-be1f-751f64eddfe7-host\") pod \"node-ca-p75rj\" (UID: \"3ef8e676-9989-483a-be1f-751f64eddfe7\") " pod="openshift-image-registry/node-ca-p75rj"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-ovnkube-config\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmc9d\" (UniqueName: \"kubernetes.io/projected/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-kube-api-access-hmc9d\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221131    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3ef8e676-9989-483a-be1f-751f64eddfe7-serviceca\") pod \"node-ca-p75rj\" (UID: \"3ef8e676-9989-483a-be1f-751f64eddfe7\") " pod="openshift-image-registry/node-ca-p75rj"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-var-lib-kubelet\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-sys\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221210    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-run-ovn\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.221611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221281    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221280    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-run\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221259    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-run-ovn\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8528fdf6-c48e-4f4d-bdc4-a685de05b47d-host-slash\") pod \"iptables-alerter-8tph9\" (UID: \"8528fdf6-c48e-4f4d-bdc4-a685de05b47d\") " pod="openshift-network-operator/iptables-alerter-8tph9"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-cni-netd\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-socket-dir-parent\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221357    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-run-multus-certs\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.221367    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221391    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8528fdf6-c48e-4f4d-bdc4-a685de05b47d-host-slash\") pod \"iptables-alerter-8tph9\" (UID: \"8528fdf6-c48e-4f4d-bdc4-a685de05b47d\") " pod="openshift-network-operator/iptables-alerter-8tph9"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221418    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-cni-netd\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.221462    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs podName:b3ef2bd6-26b1-447e-9b89-aae0cf51d91c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:30.721447677 +0000 UTC m=+4.645230332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs") pod "network-metrics-daemon-rsp6q" (UID: "b3ef2bd6-26b1-447e-9b89-aae0cf51d91c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-modprobe-d\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwkkg\" (UniqueName: \"kubernetes.io/projected/c8ba82ea-45f3-429a-83b1-d3914c12546f-kube-api-access-kwkkg\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-systemd-units\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-run-openvswitch\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-run-netns\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.222261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-conf-dir\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqrww\" (UniqueName: \"kubernetes.io/projected/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-kube-api-access-hqrww\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221575    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8528fdf6-c48e-4f4d-bdc4-a685de05b47d-iptables-alerter-script\") pod \"iptables-alerter-8tph9\" (UID: \"8528fdf6-c48e-4f4d-bdc4-a685de05b47d\") " pod="openshift-network-operator/iptables-alerter-8tph9"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221590    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-run-openvswitch\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-node-log\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-log-socket\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-sys-fs\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221668    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8ba82ea-45f3-429a-83b1-d3914c12546f-tmp\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-ovn-node-metrics-cert\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221709    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-cni-binary-copy\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-systemd-units\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-log-socket\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-node-log\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.221912    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.222018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8528fdf6-c48e-4f4d-bdc4-a685de05b47d-iptables-alerter-script\") pod \"iptables-alerter-8tph9\" (UID: \"8528fdf6-c48e-4f4d-bdc4-a685de05b47d\") " pod="openshift-network-operator/iptables-alerter-8tph9"
Sep 17 18:29:30.223018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.222238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.223779 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.222552    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:29:30.223779 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.223565    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:30.228364 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.228294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-ovn-node-metrics-cert\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.228794 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.228771    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6wn\" (UniqueName: \"kubernetes.io/projected/3ef8e676-9989-483a-be1f-751f64eddfe7-kube-api-access-gh6wn\") pod \"node-ca-p75rj\" (UID: \"3ef8e676-9989-483a-be1f-751f64eddfe7\") " pod="openshift-image-registry/node-ca-p75rj"
Sep 17 18:29:30.228969 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.228949    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wqmb\" (UniqueName: \"kubernetes.io/projected/8528fdf6-c48e-4f4d-bdc4-a685de05b47d-kube-api-access-7wqmb\") pod \"iptables-alerter-8tph9\" (UID: \"8528fdf6-c48e-4f4d-bdc4-a685de05b47d\") " pod="openshift-network-operator/iptables-alerter-8tph9"
Sep 17 18:29:30.231255 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.231234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hz88\" (UniqueName: \"kubernetes.io/projected/a76a82b7-6e16-4354-92aa-da9e6b01b704-kube-api-access-8hz88\") pod \"node-resolver-4brlg\" (UID: \"a76a82b7-6e16-4354-92aa-da9e6b01b704\") " pod="openshift-dns/node-resolver-4brlg"
Sep 17 18:29:30.231440 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.231423    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmvnq\" (UniqueName: \"kubernetes.io/projected/0755d8ba-cf82-4e44-9f74-1c53bebedf8c-kube-api-access-cmvnq\") pod \"ovnkube-node-p9vs8\" (UID: \"0755d8ba-cf82-4e44-9f74-1c53bebedf8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.231763 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.231742    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmc9d\" (UniqueName: \"kubernetes.io/projected/d6c904bd-89ec-40d5-88f4-ba4ce4d194fa-kube-api-access-hmc9d\") pod \"multus-additional-cni-plugins-5259j\" (UID: \"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa\") " pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.233309 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.233286    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vjmm\" (UniqueName: \"kubernetes.io/projected/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-kube-api-access-6vjmm\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:30.322049 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322009    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwkkg\" (UniqueName: \"kubernetes.io/projected/c8ba82ea-45f3-429a-83b1-d3914c12546f-kube-api-access-kwkkg\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.322226 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-run-netns\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.322226 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-conf-dir\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.322226 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqrww\" (UniqueName: \"kubernetes.io/projected/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-kube-api-access-hqrww\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.322226 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322131    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-sys-fs\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.322226 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8ba82ea-45f3-429a-83b1-d3914c12546f-tmp\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.322226 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-cni-binary-copy\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.322226 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-var-lib-cni-bin\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.322226 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/676e15c7-92c5-4d69-b2d4-7cc98fe94e86-konnectivity-ca\") pod \"konnectivity-agent-rwlgb\" (UID: \"676e15c7-92c5-4d69-b2d4-7cc98fe94e86\") " pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-sysconfig\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-dbus\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-var-lib-cni-multus\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-hostroot\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-lib-modules\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-tuned\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-daemon-config\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-systemd\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-socket-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zskpw\" (UniqueName: \"kubernetes.io/projected/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-kube-api-access-zskpw\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-var-lib-kubelet\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-cnibin\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-device-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.322496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322499    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-system-cni-dir\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-run-k8s-cni-cncf-io\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-kubelet-config\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/676e15c7-92c5-4d69-b2d4-7cc98fe94e86-agent-certs\") pod \"konnectivity-agent-rwlgb\" (UID: \"676e15c7-92c5-4d69-b2d4-7cc98fe94e86\") " pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-kubernetes\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h244l\" (UniqueName: \"kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l\") pod \"network-check-target-dt469\" (UID: \"24e1408d-b93a-449f-bd8e-6dfb76828d48\") " pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-registration-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-sysctl-d\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-host\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-cni-dir\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322796    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-etc-kubernetes\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322818    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-sysctl-conf\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-var-lib-kubelet\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-sys\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322934    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-run\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.322978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-socket-dir-parent\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-run-multus-certs\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-modprobe-d\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-run-netns\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323182    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-modprobe-d\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323187    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-conf-dir\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-daemon-config\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-systemd\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-var-lib-cni-bin\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-sysconfig\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323494    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-sys-fs\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323503    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-var-lib-cni-multus\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-hostroot\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323558    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-registration-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323635    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-lib-modules\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323655    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-cni-binary-copy\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323767    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-socket-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-var-lib-kubelet\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.323943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-cnibin\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323918    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-device-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.323939    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-dbus\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324009    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-system-cni-dir\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324058    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-run-k8s-cni-cncf-io\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324070    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-host\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-sysctl-d\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.324151    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324222    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.324248    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret podName:991b1e4c-1134-42a3-98d8-0c9f7a1385f2 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:30.824230219 +0000 UTC m=+4.748012861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret") pod "global-pull-secret-syncer-5v9qj" (UID: "991b1e4c-1134-42a3-98d8-0c9f7a1385f2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324272    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-kubelet-config\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-var-lib-kubelet\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324290    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-etc-kubernetes\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324333    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-kubernetes\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324340    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-sys\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-run\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.324761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324246    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/676e15c7-92c5-4d69-b2d4-7cc98fe94e86-konnectivity-ca\") pod \"konnectivity-agent-rwlgb\" (UID: \"676e15c7-92c5-4d69-b2d4-7cc98fe94e86\") " pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:30.325779 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-socket-dir-parent\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.325779 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324494    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-host-run-multus-certs\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.325779 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-multus-cni-dir\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.325779 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-sysctl-conf\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.325779 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.324857    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8ba82ea-45f3-429a-83b1-d3914c12546f-tmp\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.326154 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.326134    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c8ba82ea-45f3-429a-83b1-d3914c12546f-etc-tuned\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.327103 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.327080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/676e15c7-92c5-4d69-b2d4-7cc98fe94e86-agent-certs\") pod \"konnectivity-agent-rwlgb\" (UID: \"676e15c7-92c5-4d69-b2d4-7cc98fe94e86\") " pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:30.329769 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.329721    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:30.329769 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.329747    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:30.329769 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.329764    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h244l for pod openshift-network-diagnostics/network-check-target-dt469: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:30.330011 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.329851    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l podName:24e1408d-b93a-449f-bd8e-6dfb76828d48 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:30.829833312 +0000 UTC m=+4.753615973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h244l" (UniqueName: "kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l") pod "network-check-target-dt469" (UID: "24e1408d-b93a-449f-bd8e-6dfb76828d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:30.330524 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.330502    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwkkg\" (UniqueName: \"kubernetes.io/projected/c8ba82ea-45f3-429a-83b1-d3914c12546f-kube-api-access-kwkkg\") pod \"tuned-65s5c\" (UID: \"c8ba82ea-45f3-429a-83b1-d3914c12546f\") " pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.331170 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.331148    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqrww\" (UniqueName: \"kubernetes.io/projected/a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a-kube-api-access-hqrww\") pod \"multus-smrws\" (UID: \"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a\") " pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.332443 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.332420    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zskpw\" (UniqueName: \"kubernetes.io/projected/6070e57e-43ff-42d8-b5f2-4cdc3a77b368-kube-api-access-zskpw\") pod \"aws-ebs-csi-driver-node-h6tpz\" (UID: \"6070e57e-43ff-42d8-b5f2-4cdc3a77b368\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.405999 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.405960    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8tph9"
Sep 17 18:29:30.416612 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.416579    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4brlg"
Sep 17 18:29:30.425595 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.425569    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:30.430423 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.430401    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p75rj"
Sep 17 18:29:30.438278 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.438254    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5259j"
Sep 17 18:29:30.446088 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.446066    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz"
Sep 17 18:29:30.452915 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.452875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:30.460676 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.460649    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-65s5c"
Sep 17 18:29:30.466706 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.466678    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-smrws"
Sep 17 18:29:30.680124 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.680094    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:29:30.725604 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.725519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:30.725769 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.725648    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:30.725769 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.725716    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs podName:b3ef2bd6-26b1-447e-9b89-aae0cf51d91c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:31.725700125 +0000 UTC m=+5.649482773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs") pod "network-metrics-daemon-rsp6q" (UID: "b3ef2bd6-26b1-447e-9b89-aae0cf51d91c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:30.826205 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.826170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:30.826344 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.826312    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:30.826379 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.826373    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret podName:991b1e4c-1134-42a3-98d8-0c9f7a1385f2 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:31.826358859 +0000 UTC m=+5.750141518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret") pod "global-pull-secret-syncer-5v9qj" (UID: "991b1e4c-1134-42a3-98d8-0c9f7a1385f2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:30.927309 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:30.927272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h244l\" (UniqueName: \"kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l\") pod \"network-check-target-dt469\" (UID: \"24e1408d-b93a-449f-bd8e-6dfb76828d48\") " pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:30.927498 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.927463    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:30.927498 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.927491    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:30.927603 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.927504    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h244l for pod openshift-network-diagnostics/network-check-target-dt469: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:30.927603 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:30.927572    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l podName:24e1408d-b93a-449f-bd8e-6dfb76828d48 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:31.927553175 +0000 UTC m=+5.851335833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h244l" (UniqueName: "kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l") pod "network-check-target-dt469" (UID: "24e1408d-b93a-449f-bd8e-6dfb76828d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:31.152777 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:31.152740    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:30 +0000 UTC" deadline="2028-03-27 03:19:47.064763446 +0000 UTC"
Sep 17 18:29:31.152777 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:31.152772    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13352h50m15.911996112s"
Sep 17 18:29:31.192702 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:31.192668    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c904bd_89ec_40d5_88f4_ba4ce4d194fa.slice/crio-fe017d97e1fd3beda2289a341c706fd6c39ff98ed392b2b2de2b91c349e657ef WatchSource:0}: Error finding container fe017d97e1fd3beda2289a341c706fd6c39ff98ed392b2b2de2b91c349e657ef: Status 404 returned error can't find the container with id fe017d97e1fd3beda2289a341c706fd6c39ff98ed392b2b2de2b91c349e657ef
Sep 17 18:29:31.207820 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:31.207793    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676e15c7_92c5_4d69_b2d4_7cc98fe94e86.slice/crio-b0d7e23177905611301f0d5840153abebb5453fb01c74f29f396a3c603b233b6 WatchSource:0}: Error finding container b0d7e23177905611301f0d5840153abebb5453fb01c74f29f396a3c603b233b6: Status 404 returned error can't find the container with id b0d7e23177905611301f0d5840153abebb5453fb01c74f29f396a3c603b233b6
Sep 17 18:29:31.230934 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:29:31.228990    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ba82ea_45f3_429a_83b1_d3914c12546f.slice/crio-e629ec261a6b21a1e0eb348b522d5f6acfa0077076898bb1a7114acf6514d2c6 WatchSource:0}: Error finding container e629ec261a6b21a1e0eb348b522d5f6acfa0077076898bb1a7114acf6514d2c6: Status 404 returned error can't find the container with id e629ec261a6b21a1e0eb348b522d5f6acfa0077076898bb1a7114acf6514d2c6
Sep 17 18:29:31.733763 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:31.733661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:31.733955 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:31.733823    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:31.733955 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:31.733906    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs podName:b3ef2bd6-26b1-447e-9b89-aae0cf51d91c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:33.733870402 +0000 UTC m=+7.657653055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs") pod "network-metrics-daemon-rsp6q" (UID: "b3ef2bd6-26b1-447e-9b89-aae0cf51d91c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:31.834474 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:31.834435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:31.834703 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:31.834601    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:31.834703 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:31.834672    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret podName:991b1e4c-1134-42a3-98d8-0c9f7a1385f2 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:33.834655694 +0000 UTC m=+7.758438346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret") pod "global-pull-secret-syncer-5v9qj" (UID: "991b1e4c-1134-42a3-98d8-0c9f7a1385f2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:31.935102 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:31.935058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h244l\" (UniqueName: \"kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l\") pod \"network-check-target-dt469\" (UID: \"24e1408d-b93a-449f-bd8e-6dfb76828d48\") " pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:31.935300 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:31.935267    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:31.935300 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:31.935290    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:31.935418 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:31.935303    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h244l for pod openshift-network-diagnostics/network-check-target-dt469: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:31.935418 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:31.935363    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l podName:24e1408d-b93a-449f-bd8e-6dfb76828d48 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:33.935344698 +0000 UTC m=+7.859127352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h244l" (UniqueName: "kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l") pod "network-check-target-dt469" (UID: "24e1408d-b93a-449f-bd8e-6dfb76828d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:32.154085 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.153950    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 18:24:30 +0000 UTC" deadline="2028-03-06 13:18:06.451536107 +0000 UTC"
Sep 17 18:29:32.154085 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.154083    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12858h48m34.297458272s"
Sep 17 18:29:32.177503 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.177011    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:32.177503 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.177020    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:32.177503 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:32.177141    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:32.177503 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.177173    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:32.177503 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:32.177264    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:32.177503 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:32.177366    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:32.199330 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.199293    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p75rj" event={"ID":"3ef8e676-9989-483a-be1f-751f64eddfe7","Type":"ContainerStarted","Data":"15d8d99b94a36b2f8bbf0b91e9dec5137d1893556072dd551df8147b9868d359"}
Sep 17 18:29:32.202561 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.202485    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4brlg" event={"ID":"a76a82b7-6e16-4354-92aa-da9e6b01b704","Type":"ContainerStarted","Data":"53fbc7858ed3b3401251d2739d0acaddf8bb82a3272fa849f7e718d3cdfc571b"}
Sep 17 18:29:32.208126 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.208044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5259j" event={"ID":"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa","Type":"ContainerStarted","Data":"fe017d97e1fd3beda2289a341c706fd6c39ff98ed392b2b2de2b91c349e657ef"}
Sep 17 18:29:32.217640 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.217602    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.ec2.internal" event={"ID":"41cc877ab68985cd82a0459700bd3eff","Type":"ContainerStarted","Data":"f98cd6096afde0e4e6f2814ae693389705f38028c6aa4e9a929dd69310269805"}
Sep 17 18:29:32.220927 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.220788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-smrws" event={"ID":"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a","Type":"ContainerStarted","Data":"656bd5958c7ca6350792fc1a778d88ad399bb5e0b03b9117d7d71ea5131218c4"}
Sep 17 18:29:32.223764 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.223728    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-65s5c" event={"ID":"c8ba82ea-45f3-429a-83b1-d3914c12546f","Type":"ContainerStarted","Data":"e629ec261a6b21a1e0eb348b522d5f6acfa0077076898bb1a7114acf6514d2c6"}
Sep 17 18:29:32.231184 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.231106    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rwlgb" event={"ID":"676e15c7-92c5-4d69-b2d4-7cc98fe94e86","Type":"ContainerStarted","Data":"b0d7e23177905611301f0d5840153abebb5453fb01c74f29f396a3c603b233b6"}
Sep 17 18:29:32.233244 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.232357    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.ec2.internal" podStartSLOduration=3.232340197 podStartE2EDuration="3.232340197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:32.232058812 +0000 UTC m=+6.155841473" watchObservedRunningTime="2026-09-17 18:29:32.232340197 +0000 UTC m=+6.156122860"
Sep 17 18:29:32.239875 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.239809    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8tph9" event={"ID":"8528fdf6-c48e-4f4d-bdc4-a685de05b47d","Type":"ContainerStarted","Data":"35fab21cd12b3fc1b02458fbccb81785efa6027743a9088bd4eea363e00b6923"}
Sep 17 18:29:32.242006 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.241922    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz" event={"ID":"6070e57e-43ff-42d8-b5f2-4cdc3a77b368","Type":"ContainerStarted","Data":"6c815c3ba7a724417492494241b863617d4797fdb0627bfe21a6ae3409c2ac90"}
Sep 17 18:29:32.244067 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.243962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8" event={"ID":"0755d8ba-cf82-4e44-9f74-1c53bebedf8c","Type":"ContainerStarted","Data":"d12446b61682d08d287ad897513faa6e50530d9ebb246d769b56353b8d386ca1"}
Sep 17 18:29:32.254824 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.253950    2582 generic.go:356] "Generic (PLEG): container finished" podID="9ece2644ee37debecdc1aee17a00ea8d" containerID="200b0a3b5f2836a7af94f7389a785da068e3f15223739798acffcb7bfb2125f6" exitCode=0
Sep 17 18:29:32.254824 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:32.253992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal" event={"ID":"9ece2644ee37debecdc1aee17a00ea8d","Type":"ContainerDied","Data":"200b0a3b5f2836a7af94f7389a785da068e3f15223739798acffcb7bfb2125f6"}
Sep 17 18:29:33.269695 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:33.269440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal" event={"ID":"9ece2644ee37debecdc1aee17a00ea8d","Type":"ContainerStarted","Data":"430ceb5cfdd1cd49821a6373b2f1ad352a904437100d56d990a3ebcf2f4e4de1"}
Sep 17 18:29:33.750053 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:33.750011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:33.750223 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:33.750165    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:33.750262 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:33.750231    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs podName:b3ef2bd6-26b1-447e-9b89-aae0cf51d91c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:37.750211163 +0000 UTC m=+11.673993815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs") pod "network-metrics-daemon-rsp6q" (UID: "b3ef2bd6-26b1-447e-9b89-aae0cf51d91c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:33.851325 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:33.851285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:33.851493 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:33.851468    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:33.851572 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:33.851533    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret podName:991b1e4c-1134-42a3-98d8-0c9f7a1385f2 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:37.851513354 +0000 UTC m=+11.775295994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret") pod "global-pull-secret-syncer-5v9qj" (UID: "991b1e4c-1134-42a3-98d8-0c9f7a1385f2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:33.951862 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:33.951808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h244l\" (UniqueName: \"kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l\") pod \"network-check-target-dt469\" (UID: \"24e1408d-b93a-449f-bd8e-6dfb76828d48\") " pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:33.952064 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:33.952051    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:33.952129 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:33.952071    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:33.952129 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:33.952085    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h244l for pod openshift-network-diagnostics/network-check-target-dt469: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:33.952240 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:33.952151    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l podName:24e1408d-b93a-449f-bd8e-6dfb76828d48 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:37.952132395 +0000 UTC m=+11.875915034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h244l" (UniqueName: "kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l") pod "network-check-target-dt469" (UID: "24e1408d-b93a-449f-bd8e-6dfb76828d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:34.178092 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:34.177987    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:34.178293 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:34.178116    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:34.178474 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:34.178452    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:34.178564 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:34.178549    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:34.178629 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:34.178606    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:34.178716 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:34.178688    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:36.178900 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:36.178854    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:36.179325 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:36.178985    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:36.179325 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:36.179005    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:36.179325 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:36.179004    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:36.179325 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:36.179052    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:36.179325 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:36.179126    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:37.785027 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:37.784983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:37.785447 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:37.785150    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:37.785447 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:37.785215    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs podName:b3ef2bd6-26b1-447e-9b89-aae0cf51d91c nodeName:}" failed. No retries permitted until 2026-09-17 18:29:45.785196896 +0000 UTC m=+19.708979540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs") pod "network-metrics-daemon-rsp6q" (UID: "b3ef2bd6-26b1-447e-9b89-aae0cf51d91c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:37.885645 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:37.885608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:37.885806 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:37.885761    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:37.885846 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:37.885810    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret podName:991b1e4c-1134-42a3-98d8-0c9f7a1385f2 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:45.88579685 +0000 UTC m=+19.809579503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret") pod "global-pull-secret-syncer-5v9qj" (UID: "991b1e4c-1134-42a3-98d8-0c9f7a1385f2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:37.986900 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:37.986845    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h244l\" (UniqueName: \"kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l\") pod \"network-check-target-dt469\" (UID: \"24e1408d-b93a-449f-bd8e-6dfb76828d48\") " pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:37.987086 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:37.987051    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:37.987086 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:37.987070    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:37.987086 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:37.987083    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h244l for pod openshift-network-diagnostics/network-check-target-dt469: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:37.987225 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:37.987143    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l podName:24e1408d-b93a-449f-bd8e-6dfb76828d48 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:45.987123926 +0000 UTC m=+19.910906581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h244l" (UniqueName: "kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l") pod "network-check-target-dt469" (UID: "24e1408d-b93a-449f-bd8e-6dfb76828d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:38.193793 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:38.192548    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:38.193793 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:38.192737    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:38.193793 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:38.193250    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:38.193793 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:38.193376    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:38.193793 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:38.193423    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:38.193793 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:38.193487    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:40.176505 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:40.176463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:40.177001 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:40.176463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:40.177001 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:40.176606    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:40.177001 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:40.176633    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:40.177001 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:40.176723    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:40.177001 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:40.176786    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:42.177231 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:42.177189    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:42.177685 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:42.177189    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:42.177685 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:42.177333    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:42.177685 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:42.177435    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:42.177685 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:42.177491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:42.177685 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:42.177542    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:44.177292 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:44.177256    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:44.177797 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:44.177256    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:44.177797 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:44.177366    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:44.177797 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:44.177459    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:44.177797 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:44.177274    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:44.177797 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:44.177570    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:45.847422 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:45.847382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:45.847918 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:45.847527    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:45.847918 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:45.847605    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs podName:b3ef2bd6-26b1-447e-9b89-aae0cf51d91c nodeName:}" failed. No retries permitted until 2026-09-17 18:30:01.847584286 +0000 UTC m=+35.771366948 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs") pod "network-metrics-daemon-rsp6q" (UID: "b3ef2bd6-26b1-447e-9b89-aae0cf51d91c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:29:45.948537 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:45.948493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:45.948720 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:45.948654    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:45.948720 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:45.948715    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret podName:991b1e4c-1134-42a3-98d8-0c9f7a1385f2 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:01.9486998 +0000 UTC m=+35.872482441 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret") pod "global-pull-secret-syncer-5v9qj" (UID: "991b1e4c-1134-42a3-98d8-0c9f7a1385f2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:29:46.049054 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:46.049016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h244l\" (UniqueName: \"kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l\") pod \"network-check-target-dt469\" (UID: \"24e1408d-b93a-449f-bd8e-6dfb76828d48\") " pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:46.049221 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:46.049169    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:29:46.049221 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:46.049185    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:29:46.049221 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:46.049195    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h244l for pod openshift-network-diagnostics/network-check-target-dt469: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:46.049328 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:46.049247    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l podName:24e1408d-b93a-449f-bd8e-6dfb76828d48 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:02.04923398 +0000 UTC m=+35.973016627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h244l" (UniqueName: "kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l") pod "network-check-target-dt469" (UID: "24e1408d-b93a-449f-bd8e-6dfb76828d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:29:46.177024 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:46.176941    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:46.177178 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:46.176943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:46.177178 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:46.177085    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:46.177178 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:46.177110    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:46.177291 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:46.177226    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:46.177331 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:46.177309    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:48.177505 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:48.177460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:48.177934 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:48.177572    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:48.177934 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:48.177604    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:48.177934 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:48.177622    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:48.177934 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:48.177677    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:48.177934 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:48.177762    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:49.350214 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:49.349001    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-65s5c" podStartSLOduration=3.427174471 podStartE2EDuration="21.348979665s" totalImagesPullingTime="17.921805194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:49.348728666 +0000 UTC m=+23.272511348" watchObservedRunningTime="2026-09-17 18:29:49.348979665 +0000 UTC m=+23.272762342"
Sep 17 18:29:49.350214 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:49.349621    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.ec2.internal" podStartSLOduration=20.349605239 podStartE2EDuration="20.349605239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:33.28421612 +0000 UTC m=+7.207998789" watchObservedRunningTime="2026-09-17 18:29:49.349605239 +0000 UTC m=+23.273387901"
Sep 17 18:29:50.176528 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.176452    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:50.176792 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.176451    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:50.176792 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:50.176591    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:50.176792 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.176635    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:50.176792 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:50.176715    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:50.176792 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:50.176769    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:50.279067 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.278799    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:29:50.334575 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.334531    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p75rj" event={"ID":"3ef8e676-9989-483a-be1f-751f64eddfe7","Type":"ContainerStarted","Data":"191eedfcda83b571ca04e3ff01cd2e3dd41808447a093d547c5e9f38aed1894b"}
Sep 17 18:29:50.336100 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.336077    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4brlg" event={"ID":"a76a82b7-6e16-4354-92aa-da9e6b01b704","Type":"ContainerStarted","Data":"08f69bfe8e250506a51a006c55ada9d7043da134766774b05dff25bf0f928fe5"}
Sep 17 18:29:50.337635 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.337607    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6c904bd-89ec-40d5-88f4-ba4ce4d194fa" containerID="d805021c9aad15bb715b58304445136ed5448e2395ad587100a524ea44c77718" exitCode=0
Sep 17 18:29:50.337756 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.337687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5259j" event={"ID":"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa","Type":"ContainerDied","Data":"d805021c9aad15bb715b58304445136ed5448e2395ad587100a524ea44c77718"}
Sep 17 18:29:50.339278 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.339248    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-smrws" event={"ID":"a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a","Type":"ContainerStarted","Data":"f69e7f14923ce06d973b813882f61496229eae81bd94cf58655444339466d1ba"}
Sep 17 18:29:50.340791 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.340767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-65s5c" event={"ID":"c8ba82ea-45f3-429a-83b1-d3914c12546f","Type":"ContainerStarted","Data":"3da6206b8e6b3258bcff849ce96406e7e4b7bfcb89baa9e58390a13acdb9771b"}
Sep 17 18:29:50.342272 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.342251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rwlgb" event={"ID":"676e15c7-92c5-4d69-b2d4-7cc98fe94e86","Type":"ContainerStarted","Data":"92d1eeb68c82f1d157aa018ff71d7aa419f0b1094acb10713d3f61d36ab63cbd"}
Sep 17 18:29:50.344079 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.344061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz" event={"ID":"6070e57e-43ff-42d8-b5f2-4cdc3a77b368","Type":"ContainerStarted","Data":"646b52bbb15a33d889b252b4c611a7bfafccd45d8e4a601d30b55e6db97fc781"}
Sep 17 18:29:50.344161 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.344100    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz" event={"ID":"6070e57e-43ff-42d8-b5f2-4cdc3a77b368","Type":"ContainerStarted","Data":"202be137402b9be2e05b9761d4a19b51a16d3b4901497f0cb6dcc7beaca317a0"}
Sep 17 18:29:50.347337 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.347317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8" event={"ID":"0755d8ba-cf82-4e44-9f74-1c53bebedf8c","Type":"ContainerStarted","Data":"7e965711202c104a5a1b05e554a25306b52a0a272a8b73f5bd5613b6285588df"}
Sep 17 18:29:50.347427 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.347368    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8" event={"ID":"0755d8ba-cf82-4e44-9f74-1c53bebedf8c","Type":"ContainerStarted","Data":"34cbc99b9f09dbb936ec6186723ffe95234c923cbaf7b80062ee68331c941546"}
Sep 17 18:29:50.347427 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.347397    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8" event={"ID":"0755d8ba-cf82-4e44-9f74-1c53bebedf8c","Type":"ContainerStarted","Data":"ea773f077ffe6ac23f0f179f7535513bdbc64b49f9b2c1e0f10d8f8c9f4b74c4"}
Sep 17 18:29:50.347427 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.347424    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8" event={"ID":"0755d8ba-cf82-4e44-9f74-1c53bebedf8c","Type":"ContainerStarted","Data":"eed0105450e47dc7fa8419343a5f288837a625d0a4fbfca536b07e07354955c5"}
Sep 17 18:29:50.347572 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.347451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8" event={"ID":"0755d8ba-cf82-4e44-9f74-1c53bebedf8c","Type":"ContainerStarted","Data":"cea4fa9c4dc0cdca081702cd51dc2ff6265dcc33090baf0fd34b5bdedd51b88d"}
Sep 17 18:29:50.347572 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.347481    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8" event={"ID":"0755d8ba-cf82-4e44-9f74-1c53bebedf8c","Type":"ContainerStarted","Data":"7881ebe837e1f119daeab99e4c14f0a9cc67cf6cb3c75dffcab7bd5f329f184a"}
Sep 17 18:29:50.350441 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.350403    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p75rj" podStartSLOduration=4.402620331 podStartE2EDuration="22.350390111s" totalImagesPullingTime="17.94776978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:50.349780001 +0000 UTC m=+24.273562707" watchObservedRunningTime="2026-09-17 18:29:50.350390111 +0000 UTC m=+24.274172772"
Sep 17 18:29:50.379778 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.379727    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rwlgb" podStartSLOduration=8.468338321 podStartE2EDuration="22.379710559s" totalImagesPullingTime="13.911372238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:50.365862331 +0000 UTC m=+24.289644992" watchObservedRunningTime="2026-09-17 18:29:50.379710559 +0000 UTC m=+24.303493221"
Sep 17 18:29:50.379995 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.379920    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4brlg" podStartSLOduration=4.429060174 podStartE2EDuration="22.379914698s" totalImagesPullingTime="17.950854524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:50.379644265 +0000 UTC m=+24.303426936" watchObservedRunningTime="2026-09-17 18:29:50.379914698 +0000 UTC m=+24.303697359"
Sep 17 18:29:50.417358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.417330    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rwlgb_676e15c7-92c5-4d69-b2d4-7cc98fe94e86/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:50.418875 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.418836    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-smrws" podStartSLOduration=4.471924424 podStartE2EDuration="22.418822632s" totalImagesPullingTime="17.946898208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:50.418516313 +0000 UTC m=+24.342298974" watchObservedRunningTime="2026-09-17 18:29:50.418822632 +0000 UTC m=+24.342605318"
Sep 17 18:29:50.547029 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.546996    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rwlgb_676e15c7-92c5-4d69-b2d4-7cc98fe94e86/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:50.621207 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:50.621177    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rwlgb_676e15c7-92c5-4d69-b2d4-7cc98fe94e86/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:51.198666 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:51.198555    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:29:50.279064094Z","UUID":"e17c0d54-98f6-40cf-96c8-f4907c90a89c","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:29:51.200607 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:51.200585    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:29:51.200750 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:51.200622    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:29:51.351578 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:51.351538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8tph9" event={"ID":"8528fdf6-c48e-4f4d-bdc4-a685de05b47d","Type":"ContainerStarted","Data":"93fc0a3dd4c51e0b28f27d2e6abebfd768ed4764a09fadaef3c184df1b088eaa"}
Sep 17 18:29:51.354754 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:51.354662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz" event={"ID":"6070e57e-43ff-42d8-b5f2-4cdc3a77b368","Type":"ContainerStarted","Data":"0c2636d8622237d0c97034e628e26b54fac9fa6b48a88b95eb9a6f428d98bff2"}
Sep 17 18:29:51.385058 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:51.384980    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8tph9" podStartSLOduration=5.456168291 podStartE2EDuration="23.38496081s" totalImagesPullingTime="17.928792519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:51.366732164 +0000 UTC m=+25.290514827" watchObservedRunningTime="2026-09-17 18:29:51.38496081 +0000 UTC m=+25.308743473"
Sep 17 18:29:51.491418 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:51.491240    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rwlgb_676e15c7-92c5-4d69-b2d4-7cc98fe94e86/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:51.669277 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:51.669233    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rwlgb_676e15c7-92c5-4d69-b2d4-7cc98fe94e86/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:51.735398 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:51.735356    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rwlgb_676e15c7-92c5-4d69-b2d4-7cc98fe94e86/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:52.177228 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:52.177190    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:52.177425 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:52.177190    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:52.177425 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:52.177316    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:52.177425 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:52.177408    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:52.177593 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:52.177191    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:52.177593 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:52.177533    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:52.361319 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:52.361279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8" event={"ID":"0755d8ba-cf82-4e44-9f74-1c53bebedf8c","Type":"ContainerStarted","Data":"a52434690bfdd96b5521b79d0c171f60a17e00686efc3a06e17c06639bd2da05"}
Sep 17 18:29:53.718153 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:53.718116    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rwlgb_676e15c7-92c5-4d69-b2d4-7cc98fe94e86/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:54.177109 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.177078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:54.177109 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.177101    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:54.177318 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.177089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:54.177318 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:54.177190    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:54.177318 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:54.177252    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:54.177318 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:54.177306    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:54.370856 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.370696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8" event={"ID":"0755d8ba-cf82-4e44-9f74-1c53bebedf8c","Type":"ContainerStarted","Data":"20144f88fb6d144b49ed11d3f1e1c20753ee7cd8f4c74fdf6fdee22c1e3064e7"}
Sep 17 18:29:54.371082 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.371065    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:54.371165 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.371116    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:54.371165 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.371164    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:54.387103 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.387076    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:54.387240 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.387178    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:29:54.395918 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.395837    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6tpz" podStartSLOduration=6.7747776680000005 podStartE2EDuration="26.395814354s" totalImagesPullingTime="19.621036686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:51.38514181 +0000 UTC m=+25.308924471" watchObservedRunningTime="2026-09-17 18:29:54.395814354 +0000 UTC m=+28.319596996"
Sep 17 18:29:54.396345 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.396303    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8" podStartSLOduration=8.365944437 podStartE2EDuration="26.396282778s" totalImagesPullingTime="18.030338341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:54.39506318 +0000 UTC m=+28.318845841" watchObservedRunningTime="2026-09-17 18:29:54.396282778 +0000 UTC m=+28.320065443"
Sep 17 18:29:54.500700 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.500666    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:54.501443 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.501416    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:54.865220 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.865172    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rwlgb_676e15c7-92c5-4d69-b2d4-7cc98fe94e86/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:54.921503 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:54.921463    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rwlgb_676e15c7-92c5-4d69-b2d4-7cc98fe94e86/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.375210 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:55.375173    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6c904bd-89ec-40d5-88f4-ba4ce4d194fa" containerID="c6638be581ffb74fefaba9588d9c4a1e4d4a14500e36859fb3fb4e98b69df083" exitCode=0
Sep 17 18:29:55.375389 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:55.375252    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5259j" event={"ID":"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa","Type":"ContainerDied","Data":"c6638be581ffb74fefaba9588d9c4a1e4d4a14500e36859fb3fb4e98b69df083"}
Sep 17 18:29:55.376267 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:55.376008    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:55.376267 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:55.376098    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rwlgb"
Sep 17 18:29:56.176821 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:56.176622    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:56.177330 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:56.176628    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:56.177330 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:56.176924    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:56.177330 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:56.176686    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:56.177330 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:56.176983    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:56.177330 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:56.177082    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:56.202653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:56.202619    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5v9qj"]
Sep 17 18:29:56.205557 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:56.205532    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rsp6q"]
Sep 17 18:29:56.215665 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:56.215638    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dt469"]
Sep 17 18:29:56.377421 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:56.377386    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:56.377584 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:56.377553    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:56.377957 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:56.377938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:56.378050 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:56.378034    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:56.378096 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:56.378072    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:56.378185 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:56.378168    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:57.381500 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:57.381463    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6c904bd-89ec-40d5-88f4-ba4ce4d194fa" containerID="fa461749c1da12de666c5314d6ced9e561a0b3790c2c3fae7eff6a81b2900dd7" exitCode=0
Sep 17 18:29:57.381850 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:57.381543    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5259j" event={"ID":"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa","Type":"ContainerDied","Data":"fa461749c1da12de666c5314d6ced9e561a0b3790c2c3fae7eff6a81b2900dd7"}
Sep 17 18:29:58.177782 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:58.177743    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:29:58.177987 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:58.177837    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:29:58.177987 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:58.177843    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:29:58.177987 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:58.177872    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:29:58.177987 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:58.177961    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:29:58.178211 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:29:58.178030    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:29:59.388397 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:59.388356    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6c904bd-89ec-40d5-88f4-ba4ce4d194fa" containerID="63562e4dd097898fb1b90a74108671fd5350a41c57bfc4c020f03cd4a6cfc671" exitCode=0
Sep 17 18:29:59.388397 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:29:59.388391    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5259j" event={"ID":"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa","Type":"ContainerDied","Data":"63562e4dd097898fb1b90a74108671fd5350a41c57bfc4c020f03cd4a6cfc671"}
Sep 17 18:30:00.177367 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:00.177332    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:30:00.177595 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:00.177461    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5v9qj" podUID="991b1e4c-1134-42a3-98d8-0c9f7a1385f2"
Sep 17 18:30:00.177595 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:00.177340    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:30:00.177699 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:00.177611    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rsp6q" podUID="b3ef2bd6-26b1-447e-9b89-aae0cf51d91c"
Sep 17 18:30:00.177699 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:00.177672    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:30:00.177756 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:00.177718    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dt469" podUID="24e1408d-b93a-449f-bd8e-6dfb76828d48"
Sep 17 18:30:01.857416 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.857237    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-229.ec2.internal" event="NodeReady"
Sep 17 18:30:01.857865 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.857545    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:30:01.865374 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.865339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:30:01.865508 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:01.865475    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:30:01.865555 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:01.865531    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs podName:b3ef2bd6-26b1-447e-9b89-aae0cf51d91c nodeName:}" failed. No retries permitted until 2026-09-17 18:30:33.865516908 +0000 UTC m=+67.789299551 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs") pod "network-metrics-daemon-rsp6q" (UID: "b3ef2bd6-26b1-447e-9b89-aae0cf51d91c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:30:01.909954 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.909905    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gs8xz"]
Sep 17 18:30:01.926988 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.926954    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bmmdz"]
Sep 17 18:30:01.927172 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.927062    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:01.931242 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.931217    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:30:01.931484 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.931464    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f4hc6\""
Sep 17 18:30:01.931686 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.931673    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:30:01.937208 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.937184    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56c48b95f6-k8fmw"]
Sep 17 18:30:01.937640 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.937499    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bmmdz"
Sep 17 18:30:01.941058 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.940999    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:30:01.941058 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.941018    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:30:01.941277 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.941258    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qrnqw\""
Sep 17 18:30:01.941424 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.941408    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:30:01.953311 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.953283    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bmmdz"]
Sep 17 18:30:01.953311 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.953329    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gs8xz"]
Sep 17 18:30:01.953512 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.953355    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56c48b95f6-k8fmw"]
Sep 17 18:30:01.953512 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.953405    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:01.956226 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.956200    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:30:01.956878 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.956852    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fsz7t\""
Sep 17 18:30:01.957211 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.957094    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:30:01.957685 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.957619    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:30:01.965765 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.965736    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:30:01.966018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:01.965999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:30:01.966104 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:01.966087    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:30:01.966174 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:01.966143    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret podName:991b1e4c-1134-42a3-98d8-0c9f7a1385f2 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:33.966130279 +0000 UTC m=+67.889912921 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret") pod "global-pull-secret-syncer-5v9qj" (UID: "991b1e4c-1134-42a3-98d8-0c9f7a1385f2") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:30:02.067079 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7e34a0-916a-48d9-ab80-b75deacc2223-tmp-dir\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.067079 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0050eb62-8496-4d2a-80ba-e4204df7e439-cert\") pod \"ingress-canary-bmmdz\" (UID: \"0050eb62-8496-4d2a-80ba-e4204df7e439\") " pod="openshift-ingress-canary/ingress-canary-bmmdz"
Sep 17 18:30:02.067274 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.067274 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pnkx\" (UniqueName: \"kubernetes.io/projected/bb7e34a0-916a-48d9-ab80-b75deacc2223-kube-api-access-9pnkx\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.067274 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8hkv\" (UniqueName: \"kubernetes.io/projected/0050eb62-8496-4d2a-80ba-e4204df7e439-kube-api-access-n8hkv\") pod \"ingress-canary-bmmdz\" (UID: \"0050eb62-8496-4d2a-80ba-e4204df7e439\") " pod="openshift-ingress-canary/ingress-canary-bmmdz"
Sep 17 18:30:02.067274 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.067502 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067290    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-image-registry-private-configuration\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.067502 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h244l\" (UniqueName: \"kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l\") pod \"network-check-target-dt469\" (UID: \"24e1408d-b93a-449f-bd8e-6dfb76828d48\") " pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:30:02.067502 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-certificates\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.067502 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.067452    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:30:02.067502 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.067470    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:30:02.067502 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-trusted-ca\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.067502 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.067480    2582 projected.go:196] Error preparing data for projected volume kube-api-access-h244l for pod openshift-network-diagnostics/network-check-target-dt469: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:30:02.067502 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-installation-pull-secrets\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.067925 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067520    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33bc31b2-3547-4014-b2f8-b621f05d729e-ca-trust-extracted\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.067925 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jlq7\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-kube-api-access-7jlq7\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.067925 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.067576    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l podName:24e1408d-b93a-449f-bd8e-6dfb76828d48 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:34.067555262 +0000 UTC m=+67.991337907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h244l" (UniqueName: "kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l") pod "network-check-target-dt469" (UID: "24e1408d-b93a-449f-bd8e-6dfb76828d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:30:02.067925 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067635    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7e34a0-916a-48d9-ab80-b75deacc2223-config-volume\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.067925 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.067667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-bound-sa-token\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.168255 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.168255 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-image-registry-private-configuration\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.168490 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.168392    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:02.168490 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-certificates\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.168490 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.168472    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls podName:bb7e34a0-916a-48d9-ab80-b75deacc2223 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:02.668450645 +0000 UTC m=+36.592233298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls") pod "dns-default-gs8xz" (UID: "bb7e34a0-916a-48d9-ab80-b75deacc2223") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:02.168659 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168522    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-trusted-ca\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.168659 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-installation-pull-secrets\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.168659 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33bc31b2-3547-4014-b2f8-b621f05d729e-ca-trust-extracted\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.168816 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168658    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jlq7\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-kube-api-access-7jlq7\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.168816 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168689    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7e34a0-916a-48d9-ab80-b75deacc2223-config-volume\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.168816 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-bound-sa-token\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.168816 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7e34a0-916a-48d9-ab80-b75deacc2223-tmp-dir\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.169002 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0050eb62-8496-4d2a-80ba-e4204df7e439-cert\") pod \"ingress-canary-bmmdz\" (UID: \"0050eb62-8496-4d2a-80ba-e4204df7e439\") " pod="openshift-ingress-canary/ingress-canary-bmmdz"
Sep 17 18:30:02.169002 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.169002 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pnkx\" (UniqueName: \"kubernetes.io/projected/bb7e34a0-916a-48d9-ab80-b75deacc2223-kube-api-access-9pnkx\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.169002 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.168945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hkv\" (UniqueName: \"kubernetes.io/projected/0050eb62-8496-4d2a-80ba-e4204df7e439-kube-api-access-n8hkv\") pod \"ingress-canary-bmmdz\" (UID: \"0050eb62-8496-4d2a-80ba-e4204df7e439\") " pod="openshift-ingress-canary/ingress-canary-bmmdz"
Sep 17 18:30:02.169184 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.169166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-certificates\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.169337 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.169314    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7e34a0-916a-48d9-ab80-b75deacc2223-config-volume\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.169414 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.169314    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7e34a0-916a-48d9-ab80-b75deacc2223-tmp-dir\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.169454 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.169436    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:02.169454 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.169451    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c48b95f6-k8fmw: secret "image-registry-tls" not found
Sep 17 18:30:02.169529 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.169499    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls podName:33bc31b2-3547-4014-b2f8-b621f05d729e nodeName:}" failed. No retries permitted until 2026-09-17 18:30:02.669482735 +0000 UTC m=+36.593265375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls") pod "image-registry-56c48b95f6-k8fmw" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e") : secret "image-registry-tls" not found
Sep 17 18:30:02.169973 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.169641    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-trusted-ca\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.169973 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.169655    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33bc31b2-3547-4014-b2f8-b621f05d729e-ca-trust-extracted\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.173508 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.173482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0050eb62-8496-4d2a-80ba-e4204df7e439-cert\") pod \"ingress-canary-bmmdz\" (UID: \"0050eb62-8496-4d2a-80ba-e4204df7e439\") " pod="openshift-ingress-canary/ingress-canary-bmmdz"
Sep 17 18:30:02.173628 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.173485    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-image-registry-private-configuration\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.173904 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.173862    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-installation-pull-secrets\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.176864 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.176816    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:30:02.176971 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.176824    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:30:02.177267 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.177249    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:30:02.179227 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.179206    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bhngp\""
Sep 17 18:30:02.179311 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.179251    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:30:02.179422 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.179405    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmb9h\""
Sep 17 18:30:02.179498 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.179458    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:30:02.179600 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.179587    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:30:02.179859 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.179843    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:30:02.182782 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.182761    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jlq7\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-kube-api-access-7jlq7\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.183013 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.182990    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hkv\" (UniqueName: \"kubernetes.io/projected/0050eb62-8496-4d2a-80ba-e4204df7e439-kube-api-access-n8hkv\") pod \"ingress-canary-bmmdz\" (UID: \"0050eb62-8496-4d2a-80ba-e4204df7e439\") " pod="openshift-ingress-canary/ingress-canary-bmmdz"
Sep 17 18:30:02.183369 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.183353    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-bound-sa-token\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.184323 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.184280    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pnkx\" (UniqueName: \"kubernetes.io/projected/bb7e34a0-916a-48d9-ab80-b75deacc2223-kube-api-access-9pnkx\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.251211 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.251175    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bmmdz"
Sep 17 18:30:02.599635 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.599588    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bmmdz"]
Sep 17 18:30:02.673591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.673548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:02.673777 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:02.673611    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:02.673840 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.673823    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:02.673916 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.673904    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls podName:bb7e34a0-916a-48d9-ab80-b75deacc2223 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.673865595 +0000 UTC m=+37.597648257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls") pod "dns-default-gs8xz" (UID: "bb7e34a0-916a-48d9-ab80-b75deacc2223") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:02.674351 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.674329    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:02.674351 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.674348    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c48b95f6-k8fmw: secret "image-registry-tls" not found
Sep 17 18:30:02.674527 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:02.674398    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls podName:33bc31b2-3547-4014-b2f8-b621f05d729e nodeName:}" failed. No retries permitted until 2026-09-17 18:30:03.674385734 +0000 UTC m=+37.598168396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls") pod "image-registry-56c48b95f6-k8fmw" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e") : secret "image-registry-tls" not found
Sep 17 18:30:03.683572 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:03.683532    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:03.684428 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:03.683584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:03.684428 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:03.683678    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:03.684428 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:03.683700    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c48b95f6-k8fmw: secret "image-registry-tls" not found
Sep 17 18:30:03.684428 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:03.683712    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:03.684428 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:03.683763    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls podName:bb7e34a0-916a-48d9-ab80-b75deacc2223 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.683745751 +0000 UTC m=+39.607528403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls") pod "dns-default-gs8xz" (UID: "bb7e34a0-916a-48d9-ab80-b75deacc2223") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:03.684428 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:03.683781    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls podName:33bc31b2-3547-4014-b2f8-b621f05d729e nodeName:}" failed. No retries permitted until 2026-09-17 18:30:05.683772302 +0000 UTC m=+39.607554944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls") pod "image-registry-56c48b95f6-k8fmw" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e") : secret "image-registry-tls" not found
Sep 17 18:30:04.068321 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:04.068284    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0050eb62_8496_4d2a_80ba_e4204df7e439.slice/crio-dbc19cd7633675d5570e1c1ab5130c47344ef7db5eb9af2f013e5989dfddd59c WatchSource:0}: Error finding container dbc19cd7633675d5570e1c1ab5130c47344ef7db5eb9af2f013e5989dfddd59c: Status 404 returned error can't find the container with id dbc19cd7633675d5570e1c1ab5130c47344ef7db5eb9af2f013e5989dfddd59c
Sep 17 18:30:04.401936 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:04.401847    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bmmdz" event={"ID":"0050eb62-8496-4d2a-80ba-e4204df7e439","Type":"ContainerStarted","Data":"dbc19cd7633675d5570e1c1ab5130c47344ef7db5eb9af2f013e5989dfddd59c"}
Sep 17 18:30:05.408377 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:05.408173    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6c904bd-89ec-40d5-88f4-ba4ce4d194fa" containerID="871d521eec4ce00fc457ff9981aaf23f0b0055e7a4ecc64cdc763b41b062c2b8" exitCode=0
Sep 17 18:30:05.408377 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:05.408256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5259j" event={"ID":"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa","Type":"ContainerDied","Data":"871d521eec4ce00fc457ff9981aaf23f0b0055e7a4ecc64cdc763b41b062c2b8"}
Sep 17 18:30:05.701935 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:05.701835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:05.701935 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:05.701909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:05.702150 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:05.701991    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:05.702150 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:05.702013    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c48b95f6-k8fmw: secret "image-registry-tls" not found
Sep 17 18:30:05.702150 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:05.702027    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:05.702150 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:05.702081    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls podName:33bc31b2-3547-4014-b2f8-b621f05d729e nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.702062129 +0000 UTC m=+43.625844782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls") pod "image-registry-56c48b95f6-k8fmw" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e") : secret "image-registry-tls" not found
Sep 17 18:30:05.702150 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:05.702098    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls podName:bb7e34a0-916a-48d9-ab80-b75deacc2223 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:09.702089842 +0000 UTC m=+43.625872482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls") pod "dns-default-gs8xz" (UID: "bb7e34a0-916a-48d9-ab80-b75deacc2223") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:06.414571 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.414535    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5259j" event={"ID":"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa","Type":"ContainerStarted","Data":"b659e63694acc12531e81583ae81068430a1edb26fe98019cc2190c998e60600"}
Sep 17 18:30:06.623753 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.623719    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-khc8v"]
Sep 17 18:30:06.639695 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.639666    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-khc8v"]
Sep 17 18:30:06.639859 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.639796    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.642074 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.642049    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:30:06.642207 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.642148    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:30:06.642207 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.642174    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kvfvs\""
Sep 17 18:30:06.642207 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.642202    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:30:06.642367 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.642237    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:30:06.811911 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.811872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.812025 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.811932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf5e7039-0562-4578-a7bf-5b061c217ce1-data-volume\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.812025 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.811957    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg8jt\" (UniqueName: \"kubernetes.io/projected/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-api-access-zg8jt\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.812025 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.811995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf5e7039-0562-4578-a7bf-5b061c217ce1-crio-socket\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.812147 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.812053    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.912690 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.912654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.912690 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.912693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf5e7039-0562-4578-a7bf-5b061c217ce1-data-volume\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.912953 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.912709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg8jt\" (UniqueName: \"kubernetes.io/projected/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-api-access-zg8jt\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.912953 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.912757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf5e7039-0562-4578-a7bf-5b061c217ce1-crio-socket\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.912953 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.912792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.912953 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:06.912823    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:06.912953 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:06.912936    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm podName:bf5e7039-0562-4578-a7bf-5b061c217ce1 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:07.412914093 +0000 UTC m=+41.336696759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-khc8v" (UID: "bf5e7039-0562-4578-a7bf-5b061c217ce1") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:06.912953 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:06.912952    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:06.913207 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:06.913002    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls podName:bf5e7039-0562-4578-a7bf-5b061c217ce1 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:07.412986709 +0000 UTC m=+41.336769363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-khc8v" (UID: "bf5e7039-0562-4578-a7bf-5b061c217ce1") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:06.913207 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.913059    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf5e7039-0562-4578-a7bf-5b061c217ce1-crio-socket\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.913207 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.913144    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf5e7039-0562-4578-a7bf-5b061c217ce1-data-volume\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.925939 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.925904    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg8jt\" (UniqueName: \"kubernetes.io/projected/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-api-access-zg8jt\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:06.992380 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:06.992302    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s"]
Sep 17 18:30:07.001608 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.001584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s"
Sep 17 18:30:07.004727 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.004705    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 18:30:07.004844 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.004797    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 18:30:07.004932 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.004915    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dsjcg\""
Sep 17 18:30:07.013577 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.013552    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s"]
Sep 17 18:30:07.114263 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.114221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcjmf\" (UniqueName: \"kubernetes.io/projected/ea8a74cf-b930-4861-94f1-a80b8c54758b-kube-api-access-dcjmf\") pod \"migrator-76794cbd6c-k589s\" (UID: \"ea8a74cf-b930-4861-94f1-a80b8c54758b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s"
Sep 17 18:30:07.214876 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.214834    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcjmf\" (UniqueName: \"kubernetes.io/projected/ea8a74cf-b930-4861-94f1-a80b8c54758b-kube-api-access-dcjmf\") pod \"migrator-76794cbd6c-k589s\" (UID: \"ea8a74cf-b930-4861-94f1-a80b8c54758b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s"
Sep 17 18:30:07.224961 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.224930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcjmf\" (UniqueName: \"kubernetes.io/projected/ea8a74cf-b930-4861-94f1-a80b8c54758b-kube-api-access-dcjmf\") pod \"migrator-76794cbd6c-k589s\" (UID: \"ea8a74cf-b930-4861-94f1-a80b8c54758b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s"
Sep 17 18:30:07.313447 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.313410    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s"
Sep 17 18:30:07.416229 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.416200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:07.416630 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:07.416276    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:07.416630 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.416278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:07.416630 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:07.416339    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm podName:bf5e7039-0562-4578-a7bf-5b061c217ce1 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.416319357 +0000 UTC m=+42.340102003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-khc8v" (UID: "bf5e7039-0562-4578-a7bf-5b061c217ce1") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:07.416630 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:07.416399    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:07.416630 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:07.416445    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls podName:bf5e7039-0562-4578-a7bf-5b061c217ce1 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:08.416433336 +0000 UTC m=+42.340215974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-khc8v" (UID: "bf5e7039-0562-4578-a7bf-5b061c217ce1") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:07.418797 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.418767    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bmmdz" event={"ID":"0050eb62-8496-4d2a-80ba-e4204df7e439","Type":"ContainerStarted","Data":"be90932c2e65effb213a0cfa561f7ea6bd19ca01210449a70bf9e0e13570d0b3"}
Sep 17 18:30:07.422019 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.421994    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6c904bd-89ec-40d5-88f4-ba4ce4d194fa" containerID="b659e63694acc12531e81583ae81068430a1edb26fe98019cc2190c998e60600" exitCode=0
Sep 17 18:30:07.422149 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.422029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5259j" event={"ID":"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa","Type":"ContainerDied","Data":"b659e63694acc12531e81583ae81068430a1edb26fe98019cc2190c998e60600"}
Sep 17 18:30:07.439545 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.439442    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bmmdz" podStartSLOduration=3.798596446 podStartE2EDuration="6.439422395s" totalImagesPullingTime="2.640825949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:07.439141989 +0000 UTC m=+41.362924654" watchObservedRunningTime="2026-09-17 18:30:07.439422395 +0000 UTC m=+41.363205072"
Sep 17 18:30:07.590314 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.590238    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s"]
Sep 17 18:30:07.593346 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:07.593317    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8a74cf_b930_4861_94f1_a80b8c54758b.slice/crio-264115ab9d023666116f14e1f304a6359be468afc186202dd6a5ffc48accc087 WatchSource:0}: Error finding container 264115ab9d023666116f14e1f304a6359be468afc186202dd6a5ffc48accc087: Status 404 returned error can't find the container with id 264115ab9d023666116f14e1f304a6359be468afc186202dd6a5ffc48accc087
Sep 17 18:30:07.986907 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:07.986809    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4brlg_a76a82b7-6e16-4354-92aa-da9e6b01b704/dns-node-resolver/0.log"
Sep 17 18:30:08.423926 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.423598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:08.423926 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.423715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:08.423926 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:08.423779    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:08.423926 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:08.423829    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:08.423926 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:08.423846    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls podName:bf5e7039-0562-4578-a7bf-5b061c217ce1 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.423828624 +0000 UTC m=+44.347611267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-khc8v" (UID: "bf5e7039-0562-4578-a7bf-5b061c217ce1") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:08.423926 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:08.423897    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm podName:bf5e7039-0562-4578-a7bf-5b061c217ce1 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:10.423868631 +0000 UTC m=+44.347651273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-khc8v" (UID: "bf5e7039-0562-4578-a7bf-5b061c217ce1") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:08.429558 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.429525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5259j" event={"ID":"d6c904bd-89ec-40d5-88f4-ba4ce4d194fa","Type":"ContainerStarted","Data":"7502ad1210338a6e2024e1578054deb87b233516cd75618c3428fe52e4cf66d4"}
Sep 17 18:30:08.431102 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.431073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s" event={"ID":"ea8a74cf-b930-4861-94f1-a80b8c54758b","Type":"ContainerStarted","Data":"264115ab9d023666116f14e1f304a6359be468afc186202dd6a5ffc48accc087"}
Sep 17 18:30:08.459898 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.459820    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5259j" podStartSLOduration=11.200177993 podStartE2EDuration="40.459802875s" totalImagesPullingTime="29.259624882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:08.45865501 +0000 UTC m=+42.382437670" watchObservedRunningTime="2026-09-17 18:30:08.459802875 +0000 UTC m=+42.383585535"
Sep 17 18:30:08.661981 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.661945    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-6vs8x"]
Sep 17 18:30:08.664826 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.664810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x"
Sep 17 18:30:08.667902 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.667866    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 18:30:08.667993 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.667927    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 18:30:08.668191 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.668174    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-x6xwf\""
Sep 17 18:30:08.668270 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.668251    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 18:30:08.668337 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.668268    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 18:30:08.726906 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.726801    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-6vs8x"]
Sep 17 18:30:08.827802 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.827769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/38a66d4e-7634-4bf3-afab-6df0ce7559da-signing-key\") pod \"service-ca-55f6fc7b86-6vs8x\" (UID: \"38a66d4e-7634-4bf3-afab-6df0ce7559da\") " pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x"
Sep 17 18:30:08.828003 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.827811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/38a66d4e-7634-4bf3-afab-6df0ce7559da-signing-cabundle\") pod \"service-ca-55f6fc7b86-6vs8x\" (UID: \"38a66d4e-7634-4bf3-afab-6df0ce7559da\") " pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x"
Sep 17 18:30:08.828003 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.827838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmk94\" (UniqueName: \"kubernetes.io/projected/38a66d4e-7634-4bf3-afab-6df0ce7559da-kube-api-access-tmk94\") pod \"service-ca-55f6fc7b86-6vs8x\" (UID: \"38a66d4e-7634-4bf3-afab-6df0ce7559da\") " pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x"
Sep 17 18:30:08.928314 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.928265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/38a66d4e-7634-4bf3-afab-6df0ce7559da-signing-key\") pod \"service-ca-55f6fc7b86-6vs8x\" (UID: \"38a66d4e-7634-4bf3-afab-6df0ce7559da\") " pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x"
Sep 17 18:30:08.928476 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.928333    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/38a66d4e-7634-4bf3-afab-6df0ce7559da-signing-cabundle\") pod \"service-ca-55f6fc7b86-6vs8x\" (UID: \"38a66d4e-7634-4bf3-afab-6df0ce7559da\") " pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x"
Sep 17 18:30:08.928476 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.928374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmk94\" (UniqueName: \"kubernetes.io/projected/38a66d4e-7634-4bf3-afab-6df0ce7559da-kube-api-access-tmk94\") pod \"service-ca-55f6fc7b86-6vs8x\" (UID: \"38a66d4e-7634-4bf3-afab-6df0ce7559da\") " pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x"
Sep 17 18:30:08.929075 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.929058    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/38a66d4e-7634-4bf3-afab-6df0ce7559da-signing-cabundle\") pod \"service-ca-55f6fc7b86-6vs8x\" (UID: \"38a66d4e-7634-4bf3-afab-6df0ce7559da\") " pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x"
Sep 17 18:30:08.930908 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.930870    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/38a66d4e-7634-4bf3-afab-6df0ce7559da-signing-key\") pod \"service-ca-55f6fc7b86-6vs8x\" (UID: \"38a66d4e-7634-4bf3-afab-6df0ce7559da\") " pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x"
Sep 17 18:30:08.937843 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.937822    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmk94\" (UniqueName: \"kubernetes.io/projected/38a66d4e-7634-4bf3-afab-6df0ce7559da-kube-api-access-tmk94\") pod \"service-ca-55f6fc7b86-6vs8x\" (UID: \"38a66d4e-7634-4bf3-afab-6df0ce7559da\") " pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x"
Sep 17 18:30:08.973859 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.973819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x"
Sep 17 18:30:08.987290 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:08.987271    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p75rj_3ef8e676-9989-483a-be1f-751f64eddfe7/node-ca/0.log"
Sep 17 18:30:09.254819 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:09.254747    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-6vs8x"]
Sep 17 18:30:09.258385 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:09.258357    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a66d4e_7634_4bf3_afab_6df0ce7559da.slice/crio-fd64c9c66c67d829d4691278936e3fc88f0d29e6e0584cd47be473c07e3553bd WatchSource:0}: Error finding container fd64c9c66c67d829d4691278936e3fc88f0d29e6e0584cd47be473c07e3553bd: Status 404 returned error can't find the container with id fd64c9c66c67d829d4691278936e3fc88f0d29e6e0584cd47be473c07e3553bd
Sep 17 18:30:09.435231 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:09.435190    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s" event={"ID":"ea8a74cf-b930-4861-94f1-a80b8c54758b","Type":"ContainerStarted","Data":"bda6e74a73be2c34e3e4ee6218643f7fefaddf03f2faa58715bb3111f7550aa6"}
Sep 17 18:30:09.435750 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:09.435267    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s" event={"ID":"ea8a74cf-b930-4861-94f1-a80b8c54758b","Type":"ContainerStarted","Data":"2bb6ac36da27dc70f98ca8ca9bebe587c166e7f34fea9c787fe3f8e714d4dbcc"}
Sep 17 18:30:09.436278 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:09.436254    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x" event={"ID":"38a66d4e-7634-4bf3-afab-6df0ce7559da","Type":"ContainerStarted","Data":"fd64c9c66c67d829d4691278936e3fc88f0d29e6e0584cd47be473c07e3553bd"}
Sep 17 18:30:09.462336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:09.462273    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-k589s" podStartSLOduration=2.563034557 podStartE2EDuration="3.462257801s" totalImagesPullingTime="899.223244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:09.462033072 +0000 UTC m=+43.385815743" watchObservedRunningTime="2026-09-17 18:30:09.462257801 +0000 UTC m=+43.386040461"
Sep 17 18:30:09.735195 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:09.735165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:09.735443 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:09.735206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:09.735443 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:09.735328    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:30:09.735443 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:09.735340    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:30:09.735443 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:09.735347    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c48b95f6-k8fmw: secret "image-registry-tls" not found
Sep 17 18:30:09.735443 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:09.735403    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls podName:bb7e34a0-916a-48d9-ab80-b75deacc2223 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:17.735386292 +0000 UTC m=+51.659168935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls") pod "dns-default-gs8xz" (UID: "bb7e34a0-916a-48d9-ab80-b75deacc2223") : secret "dns-default-metrics-tls" not found
Sep 17 18:30:09.735443 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:09.735421    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls podName:33bc31b2-3547-4014-b2f8-b621f05d729e nodeName:}" failed. No retries permitted until 2026-09-17 18:30:17.735414819 +0000 UTC m=+51.659197461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls") pod "image-registry-56c48b95f6-k8fmw" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e") : secret "image-registry-tls" not found
Sep 17 18:30:09.789097 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:09.789044    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bmmdz_0050eb62-8496-4d2a-80ba-e4204df7e439/serve-healthcheck-canary/0.log"
Sep 17 18:30:10.442554 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:10.442512    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:10.443056 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:10.442600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:10.443056 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:10.442611    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:10.443056 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:10.442685    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm podName:bf5e7039-0562-4578-a7bf-5b061c217ce1 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:14.442660206 +0000 UTC m=+48.366442863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-khc8v" (UID: "bf5e7039-0562-4578-a7bf-5b061c217ce1") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:10.443056 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:10.442727    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:10.443056 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:10.442782    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls podName:bf5e7039-0562-4578-a7bf-5b061c217ce1 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:14.442766123 +0000 UTC m=+48.366548785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-khc8v" (UID: "bf5e7039-0562-4578-a7bf-5b061c217ce1") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:11.443058 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:11.443021    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x" event={"ID":"38a66d4e-7634-4bf3-afab-6df0ce7559da","Type":"ContainerStarted","Data":"e900cb137956ad4d5077973b12c528063a113ef9f19f60dfae127e1e5ba2ebee"}
Sep 17 18:30:11.463687 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:11.463567    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-6vs8x" podStartSLOduration=1.853270494 podStartE2EDuration="3.463548156s" totalImagesPullingTime="1.610277662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:11.462318096 +0000 UTC m=+45.386100757" watchObservedRunningTime="2026-09-17 18:30:11.463548156 +0000 UTC m=+45.387330819"
Sep 17 18:30:14.480433 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:14.480385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:14.480906 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:14.480496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:14.480906 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:14.480539    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:14.480906 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:14.480607    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls podName:bf5e7039-0562-4578-a7bf-5b061c217ce1 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:22.480591602 +0000 UTC m=+56.404374245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-khc8v" (UID: "bf5e7039-0562-4578-a7bf-5b061c217ce1") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:30:14.480906 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:14.480607    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:30:14.480906 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:14.480655    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm podName:bf5e7039-0562-4578-a7bf-5b061c217ce1 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:22.480643127 +0000 UTC m=+56.404425795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-khc8v" (UID: "bf5e7039-0562-4578-a7bf-5b061c217ce1") : configmap "kube-rbac-proxy" not found
Sep 17 18:30:17.806860 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:17.806816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:17.807255 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:17.806991    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:17.809315 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:17.809293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7e34a0-916a-48d9-ab80-b75deacc2223-metrics-tls\") pod \"dns-default-gs8xz\" (UID: \"bb7e34a0-916a-48d9-ab80-b75deacc2223\") " pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:17.809432 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:17.809415    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls\") pod \"image-registry-56c48b95f6-k8fmw\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") " pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:17.841452 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:17.841395    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:17.867575 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:17.867541    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:18.118296 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:18.118220    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gs8xz"]
Sep 17 18:30:18.121755 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:18.121725    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7e34a0_916a_48d9_ab80_b75deacc2223.slice/crio-435bd676bbc305a4dd8eae836700c80e0dc0257a74f360eea4cd1959c90c9a8d WatchSource:0}: Error finding container 435bd676bbc305a4dd8eae836700c80e0dc0257a74f360eea4cd1959c90c9a8d: Status 404 returned error can't find the container with id 435bd676bbc305a4dd8eae836700c80e0dc0257a74f360eea4cd1959c90c9a8d
Sep 17 18:30:18.153839 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:18.153808    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56c48b95f6-k8fmw"]
Sep 17 18:30:18.157811 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:18.157785    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bc31b2_3547_4014_b2f8_b621f05d729e.slice/crio-bcb30c9308cb32949eec53bb7f6ced1652e4755597ff04a2802b05dcc372a918 WatchSource:0}: Error finding container bcb30c9308cb32949eec53bb7f6ced1652e4755597ff04a2802b05dcc372a918: Status 404 returned error can't find the container with id bcb30c9308cb32949eec53bb7f6ced1652e4755597ff04a2802b05dcc372a918
Sep 17 18:30:18.461936 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:18.461831    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw" event={"ID":"33bc31b2-3547-4014-b2f8-b621f05d729e","Type":"ContainerStarted","Data":"c3d9be94ed0a80620d0dd18ceb0294c422b9205902440e02ec49b158c6e7989d"}
Sep 17 18:30:18.462107 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:18.461939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw" event={"ID":"33bc31b2-3547-4014-b2f8-b621f05d729e","Type":"ContainerStarted","Data":"bcb30c9308cb32949eec53bb7f6ced1652e4755597ff04a2802b05dcc372a918"}
Sep 17 18:30:18.462107 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:18.461968    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:18.463366 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:18.463331    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gs8xz" event={"ID":"bb7e34a0-916a-48d9-ab80-b75deacc2223","Type":"ContainerStarted","Data":"435bd676bbc305a4dd8eae836700c80e0dc0257a74f360eea4cd1959c90c9a8d"}
Sep 17 18:30:18.514363 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:18.514310    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw" podStartSLOduration=21.514295754 podStartE2EDuration="21.514295754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:18.513463686 +0000 UTC m=+52.437246338" watchObservedRunningTime="2026-09-17 18:30:18.514295754 +0000 UTC m=+52.438078415"
Sep 17 18:30:20.471664 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:20.471634    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gs8xz" event={"ID":"bb7e34a0-916a-48d9-ab80-b75deacc2223","Type":"ContainerStarted","Data":"ebc67937a036ec7d8cdb5239e60efcb226103dfd663f7b2bf0a00051cdfd151c"}
Sep 17 18:30:21.476774 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:21.476736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gs8xz" event={"ID":"bb7e34a0-916a-48d9-ab80-b75deacc2223","Type":"ContainerStarted","Data":"87670e3ec44702c1d805393c0e9b53767b817c8bf368096bbf72bd832567ca74"}
Sep 17 18:30:21.477250 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:21.476865    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:21.501080 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:21.500943    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gs8xz" podStartSLOduration=18.331136594 podStartE2EDuration="20.500922616s" totalImagesPullingTime="2.169786022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:21.500307757 +0000 UTC m=+55.424090440" watchObservedRunningTime="2026-09-17 18:30:21.500922616 +0000 UTC m=+55.424705299"
Sep 17 18:30:22.545569 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:22.545536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:22.546054 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:22.545690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:22.546265 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:22.546245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf5e7039-0562-4578-a7bf-5b061c217ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:22.547998 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:22.547980    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf5e7039-0562-4578-a7bf-5b061c217ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khc8v\" (UID: \"bf5e7039-0562-4578-a7bf-5b061c217ce1\") " pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:22.548907 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:22.548872    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-khc8v"
Sep 17 18:30:22.834791 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:22.834705    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-khc8v"]
Sep 17 18:30:22.838216 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:22.838180    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5e7039_0562_4578_a7bf_5b061c217ce1.slice/crio-9ed1013a95ffa04f4d1df886cf93d325f9926f9e84f080b6439ee8a58f4a7ba9 WatchSource:0}: Error finding container 9ed1013a95ffa04f4d1df886cf93d325f9926f9e84f080b6439ee8a58f4a7ba9: Status 404 returned error can't find the container with id 9ed1013a95ffa04f4d1df886cf93d325f9926f9e84f080b6439ee8a58f4a7ba9
Sep 17 18:30:23.486718 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:23.486686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khc8v" event={"ID":"bf5e7039-0562-4578-a7bf-5b061c217ce1","Type":"ContainerStarted","Data":"6286fb35a5dc00c106f3661e5caa75b1cbb47d001c4f6424d6250873f78ce42b"}
Sep 17 18:30:23.486936 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:23.486741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khc8v" event={"ID":"bf5e7039-0562-4578-a7bf-5b061c217ce1","Type":"ContainerStarted","Data":"9ed1013a95ffa04f4d1df886cf93d325f9926f9e84f080b6439ee8a58f4a7ba9"}
Sep 17 18:30:24.491753 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:24.491709    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khc8v" event={"ID":"bf5e7039-0562-4578-a7bf-5b061c217ce1","Type":"ContainerStarted","Data":"83ec429dcc2ba6511c567336e61c9c6794655247c7802dc33ad769747c9a2cf0"}
Sep 17 18:30:26.395310 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:26.395284    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p9vs8"
Sep 17 18:30:26.503004 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:26.502964    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khc8v" event={"ID":"bf5e7039-0562-4578-a7bf-5b061c217ce1","Type":"ContainerStarted","Data":"f3f3b4dd20e2e52226bcfff4857f281a5c1db67a1eb5839ca2026da7894a3ff3"}
Sep 17 18:30:26.518554 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:26.518500    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-khc8v" podStartSLOduration=17.688376977 podStartE2EDuration="20.518481517s" totalImagesPullingTime="2.83010454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:26.518371135 +0000 UTC m=+60.442153795" watchObservedRunningTime="2026-09-17 18:30:26.518481517 +0000 UTC m=+60.442264157"
Sep 17 18:30:31.001139 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.001101    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56c48b95f6-k8fmw"]
Sep 17 18:30:31.040210 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.040174    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-rgx5v"]
Sep 17 18:30:31.072415 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.072382    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-rgx5v"
Sep 17 18:30:31.084953 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.084929    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zz5tx\""
Sep 17 18:30:31.088574 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.088551    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 18:30:31.088698 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.088616    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 18:30:31.093852 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.093828    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-rgx5v"]
Sep 17 18:30:31.109152 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.109118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qwl7\" (UniqueName: \"kubernetes.io/projected/7a187b07-eeb9-4516-9c07-3a3486fa2eb1-kube-api-access-7qwl7\") pod \"downloads-6594f4c75-rgx5v\" (UID: \"7a187b07-eeb9-4516-9c07-3a3486fa2eb1\") " pod="openshift-console/downloads-6594f4c75-rgx5v"
Sep 17 18:30:31.177548 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.177515    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f95f5465-8hsbr"]
Sep 17 18:30:31.200278 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.200250    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.210473 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.210426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qwl7\" (UniqueName: \"kubernetes.io/projected/7a187b07-eeb9-4516-9c07-3a3486fa2eb1-kube-api-access-7qwl7\") pod \"downloads-6594f4c75-rgx5v\" (UID: \"7a187b07-eeb9-4516-9c07-3a3486fa2eb1\") " pod="openshift-console/downloads-6594f4c75-rgx5v"
Sep 17 18:30:31.212020 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.211996    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f95f5465-8hsbr"]
Sep 17 18:30:31.250180 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.250143    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qwl7\" (UniqueName: \"kubernetes.io/projected/7a187b07-eeb9-4516-9c07-3a3486fa2eb1-kube-api-access-7qwl7\") pod \"downloads-6594f4c75-rgx5v\" (UID: \"7a187b07-eeb9-4516-9c07-3a3486fa2eb1\") " pod="openshift-console/downloads-6594f4c75-rgx5v"
Sep 17 18:30:31.311309 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.311268    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9567c646-0644-44f7-928f-d89851032606-bound-sa-token\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.311309 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.311312    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9567c646-0644-44f7-928f-d89851032606-trusted-ca\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.311573 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.311339    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9567c646-0644-44f7-928f-d89851032606-ca-trust-extracted\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.311573 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.311363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9567c646-0644-44f7-928f-d89851032606-installation-pull-secrets\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.311573 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.311407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9567c646-0644-44f7-928f-d89851032606-registry-tls\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.311573 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.311469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9567c646-0644-44f7-928f-d89851032606-registry-certificates\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.311573 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.311509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndzpd\" (UniqueName: \"kubernetes.io/projected/9567c646-0644-44f7-928f-d89851032606-kube-api-access-ndzpd\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.311573 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.311548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9567c646-0644-44f7-928f-d89851032606-image-registry-private-configuration\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.381113 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.381069    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-rgx5v"
Sep 17 18:30:31.412262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.412235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9567c646-0644-44f7-928f-d89851032606-bound-sa-token\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.412386 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.412284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9567c646-0644-44f7-928f-d89851032606-trusted-ca\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.412386 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.412318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9567c646-0644-44f7-928f-d89851032606-ca-trust-extracted\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.412386 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.412348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9567c646-0644-44f7-928f-d89851032606-installation-pull-secrets\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.412539 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.412406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9567c646-0644-44f7-928f-d89851032606-registry-tls\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.412539 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.412435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9567c646-0644-44f7-928f-d89851032606-registry-certificates\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.412539 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.412463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndzpd\" (UniqueName: \"kubernetes.io/projected/9567c646-0644-44f7-928f-d89851032606-kube-api-access-ndzpd\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.412539 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.412502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9567c646-0644-44f7-928f-d89851032606-image-registry-private-configuration\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.412834 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.412773    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9567c646-0644-44f7-928f-d89851032606-ca-trust-extracted\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.413492 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.413464    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9567c646-0644-44f7-928f-d89851032606-registry-certificates\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.413492 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.413484    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9567c646-0644-44f7-928f-d89851032606-trusted-ca\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.415170 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.415147    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9567c646-0644-44f7-928f-d89851032606-image-registry-private-configuration\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.415270 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.415178    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9567c646-0644-44f7-928f-d89851032606-installation-pull-secrets\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.415804 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.415786    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9567c646-0644-44f7-928f-d89851032606-registry-tls\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.437952 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.437920    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndzpd\" (UniqueName: \"kubernetes.io/projected/9567c646-0644-44f7-928f-d89851032606-kube-api-access-ndzpd\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.449402 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.449349    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9567c646-0644-44f7-928f-d89851032606-bound-sa-token\") pod \"image-registry-7f95f5465-8hsbr\" (UID: \"9567c646-0644-44f7-928f-d89851032606\") " pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.481675 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.481646    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gs8xz"
Sep 17 18:30:31.509370 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.509336    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:31.694427 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.694394    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-rgx5v"]
Sep 17 18:30:31.697608 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:31.697582    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a187b07_eeb9_4516_9c07_3a3486fa2eb1.slice/crio-01ed2cb76bea8d955cbe2d9dc007fa690e879b7e1ef6c3ef5f62bdd497b12153 WatchSource:0}: Error finding container 01ed2cb76bea8d955cbe2d9dc007fa690e879b7e1ef6c3ef5f62bdd497b12153: Status 404 returned error can't find the container with id 01ed2cb76bea8d955cbe2d9dc007fa690e879b7e1ef6c3ef5f62bdd497b12153
Sep 17 18:30:31.828437 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:31.828405    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f95f5465-8hsbr"]
Sep 17 18:30:31.840565 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:31.840507    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9567c646_0644_44f7_928f_d89851032606.slice/crio-6e1804737d4cd0754278d456f8f61303f37cc819d353bbb0a6121caa9f50e9fc WatchSource:0}: Error finding container 6e1804737d4cd0754278d456f8f61303f37cc819d353bbb0a6121caa9f50e9fc: Status 404 returned error can't find the container with id 6e1804737d4cd0754278d456f8f61303f37cc819d353bbb0a6121caa9f50e9fc
Sep 17 18:30:32.530187 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:32.530141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-rgx5v" event={"ID":"7a187b07-eeb9-4516-9c07-3a3486fa2eb1","Type":"ContainerStarted","Data":"01ed2cb76bea8d955cbe2d9dc007fa690e879b7e1ef6c3ef5f62bdd497b12153"}
Sep 17 18:30:32.531978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:32.531940    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f95f5465-8hsbr" event={"ID":"9567c646-0644-44f7-928f-d89851032606","Type":"ContainerStarted","Data":"1d2d5831b7569cb420487c798fdccf1006132e2b03e03b86b121f45e56bf7d7d"}
Sep 17 18:30:32.532124 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:32.531994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f95f5465-8hsbr" event={"ID":"9567c646-0644-44f7-928f-d89851032606","Type":"ContainerStarted","Data":"6e1804737d4cd0754278d456f8f61303f37cc819d353bbb0a6121caa9f50e9fc"}
Sep 17 18:30:32.532124 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:32.532086    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:32.556447 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:32.556376    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f95f5465-8hsbr" podStartSLOduration=1.5563577579999999 podStartE2EDuration="1.556357758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:32.555775289 +0000 UTC m=+66.479557951" watchObservedRunningTime="2026-09-17 18:30:32.556357758 +0000 UTC m=+66.480140420"
Sep 17 18:30:33.079848 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.078840    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl"]
Sep 17 18:30:33.103716 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.103653    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl"]
Sep 17 18:30:33.103916 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.103783    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl"
Sep 17 18:30:33.107820 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.107791    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 18:30:33.107960 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.107806    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fnch7\""
Sep 17 18:30:33.227490 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.227452    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/df6f6550-661b-42f5-b580-3d43e73c40fc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-rdxdl\" (UID: \"df6f6550-661b-42f5-b580-3d43e73c40fc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl"
Sep 17 18:30:33.328931 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.328873    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/df6f6550-661b-42f5-b580-3d43e73c40fc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-rdxdl\" (UID: \"df6f6550-661b-42f5-b580-3d43e73c40fc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl"
Sep 17 18:30:33.331904 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.331824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/df6f6550-661b-42f5-b580-3d43e73c40fc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-rdxdl\" (UID: \"df6f6550-661b-42f5-b580-3d43e73c40fc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl"
Sep 17 18:30:33.433619 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.432354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl"
Sep 17 18:30:33.721900 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.721734    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl"]
Sep 17 18:30:33.726115 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:33.726080    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6f6550_661b_42f5_b580_3d43e73c40fc.slice/crio-18d836a9fb8edff516d086b008fd0ff08ab1853f47a4ca0cb5c939666f6cc7db WatchSource:0}: Error finding container 18d836a9fb8edff516d086b008fd0ff08ab1853f47a4ca0cb5c939666f6cc7db: Status 404 returned error can't find the container with id 18d836a9fb8edff516d086b008fd0ff08ab1853f47a4ca0cb5c939666f6cc7db
Sep 17 18:30:33.937588 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.937550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:30:33.940137 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.940114    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:30:33.951049 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:33.951020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3ef2bd6-26b1-447e-9b89-aae0cf51d91c-metrics-certs\") pod \"network-metrics-daemon-rsp6q\" (UID: \"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c\") " pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:30:34.014939 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.014845    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qmb9h\""
Sep 17 18:30:34.023097 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.023063    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rsp6q"
Sep 17 18:30:34.039075 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.039044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:30:34.043377 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.043345    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:30:34.052202 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.052173    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/991b1e4c-1134-42a3-98d8-0c9f7a1385f2-original-pull-secret\") pod \"global-pull-secret-syncer-5v9qj\" (UID: \"991b1e4c-1134-42a3-98d8-0c9f7a1385f2\") " pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:30:34.139817 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.139784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h244l\" (UniqueName: \"kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l\") pod \"network-check-target-dt469\" (UID: \"24e1408d-b93a-449f-bd8e-6dfb76828d48\") " pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:30:34.142259 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.142233    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:30:34.152847 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.152818    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:30:34.163742 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.163717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h244l\" (UniqueName: \"kubernetes.io/projected/24e1408d-b93a-449f-bd8e-6dfb76828d48-kube-api-access-h244l\") pod \"network-check-target-dt469\" (UID: \"24e1408d-b93a-449f-bd8e-6dfb76828d48\") " pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:30:34.304011 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.303975    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bhngp\""
Sep 17 18:30:34.308019 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.307988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5v9qj"
Sep 17 18:30:34.312091 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.312067    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:30:34.325283 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.325250    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rsp6q"]
Sep 17 18:30:34.331047 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:34.331006    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ef2bd6_26b1_447e_9b89_aae0cf51d91c.slice/crio-cb7017168089b5aa43fe6ac30e01eb5ed1f1fc1e495ba2839a05792742acd133 WatchSource:0}: Error finding container cb7017168089b5aa43fe6ac30e01eb5ed1f1fc1e495ba2839a05792742acd133: Status 404 returned error can't find the container with id cb7017168089b5aa43fe6ac30e01eb5ed1f1fc1e495ba2839a05792742acd133
Sep 17 18:30:34.542703 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.542660    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsp6q" event={"ID":"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c","Type":"ContainerStarted","Data":"cb7017168089b5aa43fe6ac30e01eb5ed1f1fc1e495ba2839a05792742acd133"}
Sep 17 18:30:34.544188 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.544158    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl" event={"ID":"df6f6550-661b-42f5-b580-3d43e73c40fc","Type":"ContainerStarted","Data":"18d836a9fb8edff516d086b008fd0ff08ab1853f47a4ca0cb5c939666f6cc7db"}
Sep 17 18:30:34.606393 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.606300    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5v9qj"]
Sep 17 18:30:34.608490 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:34.608443    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dt469"]
Sep 17 18:30:35.551919 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:35.550864    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl" event={"ID":"df6f6550-661b-42f5-b580-3d43e73c40fc","Type":"ContainerStarted","Data":"4a71ef034cc21ec0fbc172ffbe63af692388ed565d37c14100b364492c034435"}
Sep 17 18:30:35.552571 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:35.552478    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl"
Sep 17 18:30:35.553819 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:35.553788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5v9qj" event={"ID":"991b1e4c-1134-42a3-98d8-0c9f7a1385f2","Type":"ContainerStarted","Data":"5721169ad5116819e6fdbb6ee8e3f00b752a827ed990fc0a3fc7fae67a98db6a"}
Sep 17 18:30:35.556518 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:35.556483    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dt469" event={"ID":"24e1408d-b93a-449f-bd8e-6dfb76828d48","Type":"ContainerStarted","Data":"55d759ed237535c7589d62ac73baf74f05917f01a1f0e3ba90630f2cf665e94b"}
Sep 17 18:30:35.558857 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:35.558811    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl"
Sep 17 18:30:35.588960 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:35.588508    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-rdxdl" podStartSLOduration=1.52238633 podStartE2EDuration="2.588487329s" totalImagesPullingTime="1.066100999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:35.569481734 +0000 UTC m=+69.493264398" watchObservedRunningTime="2026-09-17 18:30:35.588487329 +0000 UTC m=+69.512269992"
Sep 17 18:30:36.143837 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.143752    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"]
Sep 17 18:30:36.147284 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.147259    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.151238 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.150600    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:30:36.151238 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.150667    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 18:30:36.151238 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.150600    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:30:36.151238 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.150867    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:30:36.151238 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.150941    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wqrg7\""
Sep 17 18:30:36.151238 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.151101    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 18:30:36.163297 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.163220    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"]
Sep 17 18:30:36.259307 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.259256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.259565 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.259342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.259565 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.259372    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.259565 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.259399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktjnl\" (UniqueName: \"kubernetes.io/projected/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-kube-api-access-ktjnl\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.361820 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.361275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.361820 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.361359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.361820 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.361386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.361820 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.361411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktjnl\" (UniqueName: \"kubernetes.io/projected/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-kube-api-access-ktjnl\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.361820 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:36.361468    2582 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 18:30:36.361820 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:30:36.361546    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-prometheus-operator-tls podName:56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7 nodeName:}" failed. No retries permitted until 2026-09-17 18:30:36.86152427 +0000 UTC m=+70.785306916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-hcr9d" (UID: "56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7") : secret "prometheus-operator-tls" not found
Sep 17 18:30:36.362287 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.362131    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.372372 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.370001    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.372372 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.371022    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktjnl\" (UniqueName: \"kubernetes.io/projected/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-kube-api-access-ktjnl\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.563745 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.563700    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsp6q" event={"ID":"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c","Type":"ContainerStarted","Data":"163bc5f59b7820901ce43c30f6bed9523889ab081052d431c3c03e2736fe769c"}
Sep 17 18:30:36.564224 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.563754    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rsp6q" event={"ID":"b3ef2bd6-26b1-447e-9b89-aae0cf51d91c","Type":"ContainerStarted","Data":"f051c88f1279ee742aea58a66e9b37ad1d97e32c53c394eaa68ddd712727d51d"}
Sep 17 18:30:36.582294 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.582222    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rsp6q" podStartSLOduration=67.230484259 podStartE2EDuration="1m8.582197864s" totalImagesPullingTime="1.351713605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:36.579310501 +0000 UTC m=+70.503093163" watchObservedRunningTime="2026-09-17 18:30:36.582197864 +0000 UTC m=+70.505980527"
Sep 17 18:30:36.866250 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.865651    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:36.873877 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:36.873844    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-hcr9d\" (UID: \"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:37.061482 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:37.061443    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"
Sep 17 18:30:37.360904 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:37.360847    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d"]
Sep 17 18:30:38.575238 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:38.575120    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d7d5a1_f6ea_4bc5_bf06_25eeb3ab67a7.slice/crio-3ea9eb468702685ad6ea80b58730820d9f2d7e895d3a5f87079628170ffab15d WatchSource:0}: Error finding container 3ea9eb468702685ad6ea80b58730820d9f2d7e895d3a5f87079628170ffab15d: Status 404 returned error can't find the container with id 3ea9eb468702685ad6ea80b58730820d9f2d7e895d3a5f87079628170ffab15d
Sep 17 18:30:39.579982 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:39.579937    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d" event={"ID":"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7","Type":"ContainerStarted","Data":"3ea9eb468702685ad6ea80b58730820d9f2d7e895d3a5f87079628170ffab15d"}
Sep 17 18:30:40.167434 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.167390    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74f5c89875-fdftv"]
Sep 17 18:30:40.170836 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.170810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.173850 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.173403    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 18:30:40.173850 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.173693    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 18:30:40.174304 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.174259    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hdv42\""
Sep 17 18:30:40.174469 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.174326    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 18:30:40.174532 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.174472    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 18:30:40.174941 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.174920    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 18:30:40.181777 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.181758    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 18:30:40.185782 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.185741    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f5c89875-fdftv"]
Sep 17 18:30:40.296005 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.295816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-oauth-config\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.296005 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.295903    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-oauth-serving-cert\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.296005 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.295953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67mjb\" (UniqueName: \"kubernetes.io/projected/5741ee57-541c-4e53-a5fa-164c75701636-kube-api-access-67mjb\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.296005 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.295982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-console-config\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.296005 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.296008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-trusted-ca-bundle\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.296364 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.296027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-serving-cert\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.296364 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.296051    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-service-ca\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.397195 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.397157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-oauth-serving-cert\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.397383 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.397207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67mjb\" (UniqueName: \"kubernetes.io/projected/5741ee57-541c-4e53-a5fa-164c75701636-kube-api-access-67mjb\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.397383 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.397248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-console-config\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.397383 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.397281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-trusted-ca-bundle\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.397383 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.397312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-serving-cert\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.397383 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.397339    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-service-ca\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.397383 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.397380    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-oauth-config\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.397999 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.397961    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-oauth-serving-cert\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.399496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.398651    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-trusted-ca-bundle\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.399496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.399344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-console-config\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.399496 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.399446    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-service-ca\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.400467 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.400440    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-oauth-config\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.401397 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.401368    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-serving-cert\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.410902 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.410804    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67mjb\" (UniqueName: \"kubernetes.io/projected/5741ee57-541c-4e53-a5fa-164c75701636-kube-api-access-67mjb\") pod \"console-74f5c89875-fdftv\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") " pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.486837 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.486476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:30:40.589037 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.588995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dt469" event={"ID":"24e1408d-b93a-449f-bd8e-6dfb76828d48","Type":"ContainerStarted","Data":"dc32dd655284ddf876845d62704c9b9624de6efe8c4cf710e0e4849589db73d0"}
Sep 17 18:30:40.589822 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.589760    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:30:40.594118 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.593615    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5v9qj" event={"ID":"991b1e4c-1134-42a3-98d8-0c9f7a1385f2","Type":"ContainerStarted","Data":"72950c51e069adb6de2c7a07b7b9b3aaafd635ea1ae82e9159f883b2f2540df1"}
Sep 17 18:30:40.609178 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.609122    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dt469" podStartSLOduration=67.129790783 podStartE2EDuration="1m12.609100701s" totalImagesPullingTime="5.479309918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:40.606849139 +0000 UTC m=+74.530631799" watchObservedRunningTime="2026-09-17 18:30:40.609100701 +0000 UTC m=+74.532883363"
Sep 17 18:30:40.628498 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.628438    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5v9qj" podStartSLOduration=67.144197673 podStartE2EDuration="1m12.628415272s" totalImagesPullingTime="5.484217599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:40.627320682 +0000 UTC m=+74.551103343" watchObservedRunningTime="2026-09-17 18:30:40.628415272 +0000 UTC m=+74.552197933"
Sep 17 18:30:40.772074 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:40.771999    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f5c89875-fdftv"]
Sep 17 18:30:41.008357 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:41.008326    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:41.142043 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:41.142002    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5741ee57_541c_4e53_a5fa_164c75701636.slice/crio-d3f5df2957c21074ec8dce68b04ba4a9c52066093db0ce7df5983067a8c1eac9 WatchSource:0}: Error finding container d3f5df2957c21074ec8dce68b04ba4a9c52066093db0ce7df5983067a8c1eac9: Status 404 returned error can't find the container with id d3f5df2957c21074ec8dce68b04ba4a9c52066093db0ce7df5983067a8c1eac9
Sep 17 18:30:41.599589 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:41.599546    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f5c89875-fdftv" event={"ID":"5741ee57-541c-4e53-a5fa-164c75701636","Type":"ContainerStarted","Data":"d3f5df2957c21074ec8dce68b04ba4a9c52066093db0ce7df5983067a8c1eac9"}
Sep 17 18:30:41.602042 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:41.601986    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d" event={"ID":"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7","Type":"ContainerStarted","Data":"13701c5d58cb7ab289305eb1896a9ff7fabcf241301d5c3f715607d0f12e8bc3"}
Sep 17 18:30:41.602228 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:41.602074    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d" event={"ID":"56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7","Type":"ContainerStarted","Data":"c0280c2ae187ec88587acf7592030b6dd7a2d38a11bcd0263140fc0e87211087"}
Sep 17 18:30:41.626553 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:41.625551    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-hcr9d" podStartSLOduration=3.014796434 podStartE2EDuration="5.625529321s" totalImagesPullingTime="2.610732887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:41.625129811 +0000 UTC m=+75.548912473" watchObservedRunningTime="2026-09-17 18:30:41.625529321 +0000 UTC m=+75.549311984"
Sep 17 18:30:43.829944 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.829905    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"]
Sep 17 18:30:43.833466 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.833444    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:43.840896 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.840844    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lzkv6\""
Sep 17 18:30:43.841262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.841242    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 18:30:43.841376 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.841260    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 18:30:43.879405 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.879319    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"]
Sep 17 18:30:43.891348 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.891308    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-66p4g"]
Sep 17 18:30:43.895336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.895311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:43.898168 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.898136    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:30:43.899130 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.898437    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-csfkx\""
Sep 17 18:30:43.899130 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.898951    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:30:43.899418 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.899327    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:30:43.935528 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.935499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e065573-72d8-4905-9f2a-b79416bd4f66-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:43.935528 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.935541    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e065573-72d8-4905-9f2a-b79416bd4f66-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:43.935757 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.935566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e065573-72d8-4905-9f2a-b79416bd4f66-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:43.935757 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:43.935672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzpzb\" (UniqueName: \"kubernetes.io/projected/5e065573-72d8-4905-9f2a-b79416bd4f66-kube-api-access-tzpzb\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:44.037180 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037142    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5081b60-bb6e-49d2-81c3-a1bba090bf97-root\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.037378 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037192    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbfbg\" (UniqueName: \"kubernetes.io/projected/a5081b60-bb6e-49d2-81c3-a1bba090bf97-kube-api-access-fbfbg\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.037378 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037236    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzpzb\" (UniqueName: \"kubernetes.io/projected/5e065573-72d8-4905-9f2a-b79416bd4f66-kube-api-access-tzpzb\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:44.037378 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5081b60-bb6e-49d2-81c3-a1bba090bf97-sys\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.037378 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-tls\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.037378 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037356    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-wtmp\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.037642 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5081b60-bb6e-49d2-81c3-a1bba090bf97-metrics-client-ca\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.037642 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-accelerators-collector-config\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.037642 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e065573-72d8-4905-9f2a-b79416bd4f66-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:44.037642 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-textfile\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.037642 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037564    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e065573-72d8-4905-9f2a-b79416bd4f66-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:44.037642 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e065573-72d8-4905-9f2a-b79416bd4f66-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:44.037642 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.037637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.038379 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.038354    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e065573-72d8-4905-9f2a-b79416bd4f66-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:44.040466 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.040447    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e065573-72d8-4905-9f2a-b79416bd4f66-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:44.040592 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.040566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e065573-72d8-4905-9f2a-b79416bd4f66-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:44.050175 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.050149    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzpzb\" (UniqueName: \"kubernetes.io/projected/5e065573-72d8-4905-9f2a-b79416bd4f66-kube-api-access-tzpzb\") pod \"openshift-state-metrics-86c5bc79d6-k44fb\" (UID: \"5e065573-72d8-4905-9f2a-b79416bd4f66\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:44.138122 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138034    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5081b60-bb6e-49d2-81c3-a1bba090bf97-sys\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138122 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-tls\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138339 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5081b60-bb6e-49d2-81c3-a1bba090bf97-sys\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138339 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-wtmp\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138339 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5081b60-bb6e-49d2-81c3-a1bba090bf97-metrics-client-ca\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138339 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-accelerators-collector-config\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138533 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-textfile\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138533 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-wtmp\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138533 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138533 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5081b60-bb6e-49d2-81c3-a1bba090bf97-root\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138533 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbfbg\" (UniqueName: \"kubernetes.io/projected/a5081b60-bb6e-49d2-81c3-a1bba090bf97-kube-api-access-fbfbg\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138770 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5081b60-bb6e-49d2-81c3-a1bba090bf97-root\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138903 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138737    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-textfile\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.138974 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.138959    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-accelerators-collector-config\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.139090 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.139042    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5081b60-bb6e-49d2-81c3-a1bba090bf97-metrics-client-ca\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.141370 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.141346    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-tls\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.141714 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.141689    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5081b60-bb6e-49d2-81c3-a1bba090bf97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.145333 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.145296    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"
Sep 17 18:30:44.148268 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.148246    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbfbg\" (UniqueName: \"kubernetes.io/projected/a5081b60-bb6e-49d2-81c3-a1bba090bf97-kube-api-access-fbfbg\") pod \"node-exporter-66p4g\" (UID: \"a5081b60-bb6e-49d2-81c3-a1bba090bf97\") " pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.208259 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.208016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-66p4g"
Sep 17 18:30:44.847028 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.846990    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:30:44.852004 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.851458    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.856019 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.855940    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 18:30:44.856019 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.855960    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 18:30:44.856235 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.856037    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 18:30:44.856566 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.856545    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 18:30:44.856691 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.856588    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 18:30:44.856691 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.856608    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 18:30:44.856691 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.856547    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 18:30:44.856853 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.856545    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 18:30:44.856853 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.856847    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 18:30:44.857103 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.857088    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-x7kzt\""
Sep 17 18:30:44.874297 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.874264    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:30:44.948010 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.947970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948010 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-web-config\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948260 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948050    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948260 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948260 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948205    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948260 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948239    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767jb\" (UniqueName: \"kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-kube-api-access-767jb\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948446 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948446 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948339    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-config-volume\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948446 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-config-out\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948446 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948446 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948675 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:44.948675 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:44.948550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049350 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049502 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049363    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049502 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-web-config\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049502 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049502 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049437    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049502 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049692 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767jb\" (UniqueName: \"kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-kube-api-access-767jb\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049692 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049692 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-config-volume\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049692 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-config-out\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049692 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049631    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.049692 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.050009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049698    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.050009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.049754    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.050850 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.050821    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.051057 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.051027    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.053270 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.052959    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-config-out\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.053696 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.053441    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.054606 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.054578    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-web-config\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.055560 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.055506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.055663 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.055587    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.055727 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.055678    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.055817 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.055791    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.055871 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.055815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-config-volume\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.056070 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.056053    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.064388 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.064363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-767jb\" (UniqueName: \"kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-kube-api-access-767jb\") pod \"alertmanager-main-0\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:45.163585 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:45.163503    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:30:48.295470 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.295429    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8f958cdc9-clxc2"]
Sep 17 18:30:48.298869 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.298844    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.302761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.302268    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dmdlf\""
Sep 17 18:30:48.302761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.302304    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 18:30:48.302761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.302359    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 18:30:48.302761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.302313    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 18:30:48.302761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.302605    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 18:30:48.302761 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.302733    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dvjgq9pdtgfer\""
Sep 17 18:30:48.310341 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.310279    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8f958cdc9-clxc2"]
Sep 17 18:30:48.378875 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.378829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/59c97eda-8507-4d0d-ac71-16d8dee61104-metrics-server-audit-profiles\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.379096 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.378925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/59c97eda-8507-4d0d-ac71-16d8dee61104-secret-metrics-server-client-certs\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.379096 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.378982    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/59c97eda-8507-4d0d-ac71-16d8dee61104-secret-metrics-server-tls\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.379096 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.379045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c97eda-8507-4d0d-ac71-16d8dee61104-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.379253 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.379114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68dw4\" (UniqueName: \"kubernetes.io/projected/59c97eda-8507-4d0d-ac71-16d8dee61104-kube-api-access-68dw4\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.379253 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.379148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/59c97eda-8507-4d0d-ac71-16d8dee61104-client-ca-bundle\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.379253 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.379183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/59c97eda-8507-4d0d-ac71-16d8dee61104-audit-log\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.480253 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.480215    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/59c97eda-8507-4d0d-ac71-16d8dee61104-audit-log\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.480416 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.480288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/59c97eda-8507-4d0d-ac71-16d8dee61104-metrics-server-audit-profiles\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.480416 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.480326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/59c97eda-8507-4d0d-ac71-16d8dee61104-secret-metrics-server-client-certs\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.480416 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.480358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/59c97eda-8507-4d0d-ac71-16d8dee61104-secret-metrics-server-tls\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.480416 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.480383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c97eda-8507-4d0d-ac71-16d8dee61104-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.480416 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.480414    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68dw4\" (UniqueName: \"kubernetes.io/projected/59c97eda-8507-4d0d-ac71-16d8dee61104-kube-api-access-68dw4\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.480670 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.480441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/59c97eda-8507-4d0d-ac71-16d8dee61104-client-ca-bundle\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.480733 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.480708    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/59c97eda-8507-4d0d-ac71-16d8dee61104-audit-log\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.481413 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.481385    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59c97eda-8507-4d0d-ac71-16d8dee61104-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.481627 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.481605    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/59c97eda-8507-4d0d-ac71-16d8dee61104-metrics-server-audit-profiles\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.483641 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.483613    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/59c97eda-8507-4d0d-ac71-16d8dee61104-secret-metrics-server-tls\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.483748 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.483703    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/59c97eda-8507-4d0d-ac71-16d8dee61104-client-ca-bundle\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.483911 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.483865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/59c97eda-8507-4d0d-ac71-16d8dee61104-secret-metrics-server-client-certs\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.496500 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.496472    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68dw4\" (UniqueName: \"kubernetes.io/projected/59c97eda-8507-4d0d-ac71-16d8dee61104-kube-api-access-68dw4\") pod \"metrics-server-8f958cdc9-clxc2\" (UID: \"59c97eda-8507-4d0d-ac71-16d8dee61104\") " pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:48.612767 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:48.612672    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:30:50.070508 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:50.070477    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f5c89875-fdftv"]
Sep 17 18:30:52.846934 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:52.846877    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5081b60_bb6e_49d2_81c3_a1bba090bf97.slice/crio-fb564e038a1cdb76b09766ab12bf45afdef7f65f918cf6a630182542dde62bf4 WatchSource:0}: Error finding container fb564e038a1cdb76b09766ab12bf45afdef7f65f918cf6a630182542dde62bf4: Status 404 returned error can't find the container with id fb564e038a1cdb76b09766ab12bf45afdef7f65f918cf6a630182542dde62bf4
Sep 17 18:30:53.131621 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.131326    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:30:53.139439 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.139415    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb"]
Sep 17 18:30:53.142758 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.142733    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8f958cdc9-clxc2"]
Sep 17 18:30:53.144166 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:53.144014    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e065573_72d8_4905_9f2a_b79416bd4f66.slice/crio-f36d94d5fc789f33459f781ea50b381d2899ba738e8e9285b5d625e1844f681c WatchSource:0}: Error finding container f36d94d5fc789f33459f781ea50b381d2899ba738e8e9285b5d625e1844f681c: Status 404 returned error can't find the container with id f36d94d5fc789f33459f781ea50b381d2899ba738e8e9285b5d625e1844f681c
Sep 17 18:30:53.146262 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:30:53.146235    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c97eda_8507_4d0d_ac71_16d8dee61104.slice/crio-e4655f8379679ccb4383aaae39fa42d0d0af70db825a52b32362bd07d28f08e4 WatchSource:0}: Error finding container e4655f8379679ccb4383aaae39fa42d0d0af70db825a52b32362bd07d28f08e4: Status 404 returned error can't find the container with id e4655f8379679ccb4383aaae39fa42d0d0af70db825a52b32362bd07d28f08e4
Sep 17 18:30:53.543093 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.543014    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f95f5465-8hsbr"
Sep 17 18:30:53.661263 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.661194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66p4g" event={"ID":"a5081b60-bb6e-49d2-81c3-a1bba090bf97","Type":"ContainerStarted","Data":"fb564e038a1cdb76b09766ab12bf45afdef7f65f918cf6a630182542dde62bf4"}
Sep 17 18:30:53.665343 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.665196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerStarted","Data":"dd2fea4d79c6fdf62390c4a53bc8fea85eed4138f4e3d7a9096eb9358395a7fe"}
Sep 17 18:30:53.668667 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.668001    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-rgx5v" event={"ID":"7a187b07-eeb9-4516-9c07-3a3486fa2eb1","Type":"ContainerStarted","Data":"f8f4e497de86f62e8705592dc2e80c67023a659091a90e4bd992f5e06a1ac9ed"}
Sep 17 18:30:53.669170 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.669092    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-rgx5v"
Sep 17 18:30:53.673425 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.673225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f5c89875-fdftv" event={"ID":"5741ee57-541c-4e53-a5fa-164c75701636","Type":"ContainerStarted","Data":"114aaf91e84c4953b5ebdc54f20f5bab01dfac819f51f2230e59b3c39621d06f"}
Sep 17 18:30:53.674603 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.674575    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-rgx5v"
Sep 17 18:30:53.675620 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.675588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2" event={"ID":"59c97eda-8507-4d0d-ac71-16d8dee61104","Type":"ContainerStarted","Data":"e4655f8379679ccb4383aaae39fa42d0d0af70db825a52b32362bd07d28f08e4"}
Sep 17 18:30:53.680411 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.680229    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb" event={"ID":"5e065573-72d8-4905-9f2a-b79416bd4f66","Type":"ContainerStarted","Data":"0cc87375b705387a81de77d313d0825144cba7b4a16075fb2fd73c44755d30f9"}
Sep 17 18:30:53.680411 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.680307    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb" event={"ID":"5e065573-72d8-4905-9f2a-b79416bd4f66","Type":"ContainerStarted","Data":"213681bc8162d7fcad5fb8bbbd02bbb0ba307c345a95f73052dc08c9e6f78832"}
Sep 17 18:30:53.680411 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.680335    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb" event={"ID":"5e065573-72d8-4905-9f2a-b79416bd4f66","Type":"ContainerStarted","Data":"f36d94d5fc789f33459f781ea50b381d2899ba738e8e9285b5d625e1844f681c"}
Sep 17 18:30:53.691148 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.691088    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-rgx5v" podStartSLOduration=1.5492998550000001 podStartE2EDuration="22.691067789s" totalImagesPullingTime="21.141767934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:53.690381362 +0000 UTC m=+87.614164028" watchObservedRunningTime="2026-09-17 18:30:53.691067789 +0000 UTC m=+87.614850450"
Sep 17 18:30:53.722782 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:53.722710    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74f5c89875-fdftv" podStartSLOduration=1.941227076 podStartE2EDuration="13.722685816s" totalImagesPullingTime="11.78145874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:53.721208886 +0000 UTC m=+87.644991551" watchObservedRunningTime="2026-09-17 18:30:53.722685816 +0000 UTC m=+87.646468476"
Sep 17 18:30:55.696288 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:55.696179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2" event={"ID":"59c97eda-8507-4d0d-ac71-16d8dee61104","Type":"ContainerStarted","Data":"e2d0cfd4f4c731f597870f14af4308f671a73c36d2b032d97566444be047bc73"}
Sep 17 18:30:55.699931 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:55.698951    2582 generic.go:356] "Generic (PLEG): container finished" podID="a5081b60-bb6e-49d2-81c3-a1bba090bf97" containerID="32ce36f9536db818dcc54fa155ba2c20d838c0949ff97e4540e0cba567e9fcf5" exitCode=0
Sep 17 18:30:55.700160 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:55.700094    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66p4g" event={"ID":"a5081b60-bb6e-49d2-81c3-a1bba090bf97","Type":"ContainerDied","Data":"32ce36f9536db818dcc54fa155ba2c20d838c0949ff97e4540e0cba567e9fcf5"}
Sep 17 18:30:56.020450 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.020342    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw" podUID="33bc31b2-3547-4014-b2f8-b621f05d729e" containerName="registry" containerID="cri-o://c3d9be94ed0a80620d0dd18ceb0294c422b9205902440e02ec49b158c6e7989d" gracePeriod=30
Sep 17 18:30:56.705507 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.705460    2582 generic.go:356] "Generic (PLEG): container finished" podID="33bc31b2-3547-4014-b2f8-b621f05d729e" containerID="c3d9be94ed0a80620d0dd18ceb0294c422b9205902440e02ec49b158c6e7989d" exitCode=0
Sep 17 18:30:56.706419 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.706345    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw" event={"ID":"33bc31b2-3547-4014-b2f8-b621f05d729e","Type":"ContainerDied","Data":"c3d9be94ed0a80620d0dd18ceb0294c422b9205902440e02ec49b158c6e7989d"}
Sep 17 18:30:56.866847 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.866810    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:56.889693 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.889640    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2" podStartSLOduration=6.455896987 podStartE2EDuration="8.889617953s" totalImagesPullingTime="2.433720966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:56.731169608 +0000 UTC m=+90.654952269" watchObservedRunningTime="2026-09-17 18:30:56.889617953 +0000 UTC m=+90.813400616"
Sep 17 18:30:56.974326 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.974296    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-certificates\") pod \"33bc31b2-3547-4014-b2f8-b621f05d729e\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") "
Sep 17 18:30:56.974460 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.974355    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33bc31b2-3547-4014-b2f8-b621f05d729e-ca-trust-extracted\") pod \"33bc31b2-3547-4014-b2f8-b621f05d729e\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") "
Sep 17 18:30:56.974460 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.974386    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jlq7\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-kube-api-access-7jlq7\") pod \"33bc31b2-3547-4014-b2f8-b621f05d729e\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") "
Sep 17 18:30:56.974460 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.974421    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-image-registry-private-configuration\") pod \"33bc31b2-3547-4014-b2f8-b621f05d729e\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") "
Sep 17 18:30:56.974626 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.974457    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-bound-sa-token\") pod \"33bc31b2-3547-4014-b2f8-b621f05d729e\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") "
Sep 17 18:30:56.974626 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.974502    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls\") pod \"33bc31b2-3547-4014-b2f8-b621f05d729e\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") "
Sep 17 18:30:56.974626 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.974558    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-trusted-ca\") pod \"33bc31b2-3547-4014-b2f8-b621f05d729e\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") "
Sep 17 18:30:56.974626 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.974582    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-installation-pull-secrets\") pod \"33bc31b2-3547-4014-b2f8-b621f05d729e\" (UID: \"33bc31b2-3547-4014-b2f8-b621f05d729e\") "
Sep 17 18:30:56.974985 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.974741    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "33bc31b2-3547-4014-b2f8-b621f05d729e" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:56.974985 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.974869    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-certificates\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:30:56.977638 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.977606    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-kube-api-access-7jlq7" (OuterVolumeSpecName: "kube-api-access-7jlq7") pod "33bc31b2-3547-4014-b2f8-b621f05d729e" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e"). InnerVolumeSpecName "kube-api-access-7jlq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:56.977756 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.977611    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "33bc31b2-3547-4014-b2f8-b621f05d729e" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:56.977875 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.977843    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "33bc31b2-3547-4014-b2f8-b621f05d729e" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:56.978212 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.978167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "33bc31b2-3547-4014-b2f8-b621f05d729e" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:56.980000 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.979969    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "33bc31b2-3547-4014-b2f8-b621f05d729e" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:56.980321 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.980292    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "33bc31b2-3547-4014-b2f8-b621f05d729e" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:56.985629 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:56.985595    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33bc31b2-3547-4014-b2f8-b621f05d729e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "33bc31b2-3547-4014-b2f8-b621f05d729e" (UID: "33bc31b2-3547-4014-b2f8-b621f05d729e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:57.075448 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.075272    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-installation-pull-secrets\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:30:57.075448 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.075302    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/33bc31b2-3547-4014-b2f8-b621f05d729e-ca-trust-extracted\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:30:57.075448 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.075316    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jlq7\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-kube-api-access-7jlq7\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:30:57.075448 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.075331    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/33bc31b2-3547-4014-b2f8-b621f05d729e-image-registry-private-configuration\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:30:57.075448 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.075346    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-bound-sa-token\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:30:57.075448 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.075358    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/33bc31b2-3547-4014-b2f8-b621f05d729e-registry-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:30:57.075448 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.075370    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/33bc31b2-3547-4014-b2f8-b621f05d729e-trusted-ca\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:30:57.714299 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.712632    2582 generic.go:356] "Generic (PLEG): container finished" podID="03e99b64-9872-422f-af14-319e046351fd" containerID="4e9cf9ee90815b731825e0d100df21cbf4e07219049a07cfb4e0f9f81790bddc" exitCode=0
Sep 17 18:30:57.714299 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.712773    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerDied","Data":"4e9cf9ee90815b731825e0d100df21cbf4e07219049a07cfb4e0f9f81790bddc"}
Sep 17 18:30:57.717115 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.717094    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw"
Sep 17 18:30:57.717228 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.717088    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c48b95f6-k8fmw" event={"ID":"33bc31b2-3547-4014-b2f8-b621f05d729e","Type":"ContainerDied","Data":"bcb30c9308cb32949eec53bb7f6ced1652e4755597ff04a2802b05dcc372a918"}
Sep 17 18:30:57.717303 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.717228    2582 scope.go:117] "RemoveContainer" containerID="c3d9be94ed0a80620d0dd18ceb0294c422b9205902440e02ec49b158c6e7989d"
Sep 17 18:30:57.721605 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.721572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb" event={"ID":"5e065573-72d8-4905-9f2a-b79416bd4f66","Type":"ContainerStarted","Data":"1ce2ccf299170fbbd4b962a06bcbddc406d9eb52178c54cd875ddc7dde761efe"}
Sep 17 18:30:57.725129 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.725034    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66p4g" event={"ID":"a5081b60-bb6e-49d2-81c3-a1bba090bf97","Type":"ContainerStarted","Data":"babf2f4f98cb50c0c90e1b76d8e2088857d815f4847941ec66a2b2ee030a9645"}
Sep 17 18:30:57.725129 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.725091    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66p4g" event={"ID":"a5081b60-bb6e-49d2-81c3-a1bba090bf97","Type":"ContainerStarted","Data":"8a5a6ded2e57b805097eb30717158eedd9cbe7d06afafb2115bc8c86105c6bb4"}
Sep 17 18:30:57.785820 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.785761    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-66p4g" podStartSLOduration=12.268703132 podStartE2EDuration="14.785741844s" totalImagesPullingTime="2.423845222s" totalInitContainerRuntime="93.19349ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:57.765734409 +0000 UTC m=+91.689517071" watchObservedRunningTime="2026-09-17 18:30:57.785741844 +0000 UTC m=+91.709524541"
Sep 17 18:30:57.787358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.787316    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56c48b95f6-k8fmw"]
Sep 17 18:30:57.788864 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:57.788832    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56c48b95f6-k8fmw"]
Sep 17 18:30:58.187442 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:30:58.187403    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33bc31b2-3547-4014-b2f8-b621f05d729e" path="/var/lib/kubelet/pods/33bc31b2-3547-4014-b2f8-b621f05d729e/volumes"
Sep 17 18:31:00.487377 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:00.487336    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:31:00.748644 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:00.748558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerStarted","Data":"016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e"}
Sep 17 18:31:00.748812 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:00.748645    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerStarted","Data":"d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00"}
Sep 17 18:31:00.748812 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:00.748684    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerStarted","Data":"7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743"}
Sep 17 18:31:00.748812 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:00.748718    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerStarted","Data":"2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956"}
Sep 17 18:31:00.748812 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:00.748751    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerStarted","Data":"51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e"}
Sep 17 18:31:02.765073 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:02.765019    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerStarted","Data":"de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4"}
Sep 17 18:31:02.795958 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:02.795875    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-k44fb" podStartSLOduration=16.232168948 podStartE2EDuration="19.795854318s" totalImagesPullingTime="3.56368537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:57.812186364 +0000 UTC m=+91.735969041" watchObservedRunningTime="2026-09-17 18:31:02.795854318 +0000 UTC m=+96.719636992"
Sep 17 18:31:02.796685 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:02.796656    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=11.724013205 podStartE2EDuration="18.796645132s" totalImagesPullingTime="7.063443462s" totalInitContainerRuntime="9.188465ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:02.794304073 +0000 UTC m=+96.718086736" watchObservedRunningTime="2026-09-17 18:31:02.796645132 +0000 UTC m=+96.720427792"
Sep 17 18:31:08.613346 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:08.613303    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:31:08.613799 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:08.613368    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:31:12.610664 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:12.610630    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dt469"
Sep 17 18:31:17.300652 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:17.300617    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bmmdz_0050eb62-8496-4d2a-80ba-e4204df7e439/serve-healthcheck-canary/0.log"
Sep 17 18:31:18.703537 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:18.703473    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74f5c89875-fdftv" podUID="5741ee57-541c-4e53-a5fa-164c75701636" containerName="console" containerID="cri-o://114aaf91e84c4953b5ebdc54f20f5bab01dfac819f51f2230e59b3c39621d06f" gracePeriod=15
Sep 17 18:31:18.831254 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:18.831215    2582 generic.go:356] "Generic (PLEG): container finished" podID="5741ee57-541c-4e53-a5fa-164c75701636" containerID="114aaf91e84c4953b5ebdc54f20f5bab01dfac819f51f2230e59b3c39621d06f" exitCode=0
Sep 17 18:31:18.831420 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:18.831293    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f5c89875-fdftv" event={"ID":"5741ee57-541c-4e53-a5fa-164c75701636","Type":"ContainerDied","Data":"114aaf91e84c4953b5ebdc54f20f5bab01dfac819f51f2230e59b3c39621d06f"}
Sep 17 18:31:18.967645 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:18.967570    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:31:19.057945 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.057874    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-trusted-ca-bundle\") pod \"5741ee57-541c-4e53-a5fa-164c75701636\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") "
Sep 17 18:31:19.058148 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.057957    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67mjb\" (UniqueName: \"kubernetes.io/projected/5741ee57-541c-4e53-a5fa-164c75701636-kube-api-access-67mjb\") pod \"5741ee57-541c-4e53-a5fa-164c75701636\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") "
Sep 17 18:31:19.058148 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.057992    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-oauth-config\") pod \"5741ee57-541c-4e53-a5fa-164c75701636\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") "
Sep 17 18:31:19.058148 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058029    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-console-config\") pod \"5741ee57-541c-4e53-a5fa-164c75701636\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") "
Sep 17 18:31:19.058292 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058214    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-service-ca\") pod \"5741ee57-541c-4e53-a5fa-164c75701636\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") "
Sep 17 18:31:19.058292 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058269    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-serving-cert\") pod \"5741ee57-541c-4e53-a5fa-164c75701636\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") "
Sep 17 18:31:19.058391 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058336    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-oauth-serving-cert\") pod \"5741ee57-541c-4e53-a5fa-164c75701636\" (UID: \"5741ee57-541c-4e53-a5fa-164c75701636\") "
Sep 17 18:31:19.058866 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058581    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5741ee57-541c-4e53-a5fa-164c75701636" (UID: "5741ee57-541c-4e53-a5fa-164c75701636"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:19.058866 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058630    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-console-config" (OuterVolumeSpecName: "console-config") pod "5741ee57-541c-4e53-a5fa-164c75701636" (UID: "5741ee57-541c-4e53-a5fa-164c75701636"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:19.058866 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058689    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-service-ca" (OuterVolumeSpecName: "service-ca") pod "5741ee57-541c-4e53-a5fa-164c75701636" (UID: "5741ee57-541c-4e53-a5fa-164c75701636"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:19.058866 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058720    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5741ee57-541c-4e53-a5fa-164c75701636" (UID: "5741ee57-541c-4e53-a5fa-164c75701636"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:19.058866 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058793    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-oauth-serving-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:31:19.058866 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058814    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-trusted-ca-bundle\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:31:19.058866 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058829    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-console-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:31:19.058866 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.058843    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5741ee57-541c-4e53-a5fa-164c75701636-service-ca\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:31:19.060723 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.060700    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5741ee57-541c-4e53-a5fa-164c75701636" (UID: "5741ee57-541c-4e53-a5fa-164c75701636"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:19.060907 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.060871    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5741ee57-541c-4e53-a5fa-164c75701636" (UID: "5741ee57-541c-4e53-a5fa-164c75701636"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:19.061031 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.060918    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5741ee57-541c-4e53-a5fa-164c75701636-kube-api-access-67mjb" (OuterVolumeSpecName: "kube-api-access-67mjb") pod "5741ee57-541c-4e53-a5fa-164c75701636" (UID: "5741ee57-541c-4e53-a5fa-164c75701636"). InnerVolumeSpecName "kube-api-access-67mjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:19.159816 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.159777    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67mjb\" (UniqueName: \"kubernetes.io/projected/5741ee57-541c-4e53-a5fa-164c75701636-kube-api-access-67mjb\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:31:19.159816 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.159810    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-oauth-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:31:19.159816 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.159820    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5741ee57-541c-4e53-a5fa-164c75701636-console-serving-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:31:19.836688 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.836661    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f5c89875-fdftv"
Sep 17 18:31:19.836688 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.836682    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f5c89875-fdftv" event={"ID":"5741ee57-541c-4e53-a5fa-164c75701636","Type":"ContainerDied","Data":"d3f5df2957c21074ec8dce68b04ba4a9c52066093db0ce7df5983067a8c1eac9"}
Sep 17 18:31:19.837313 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.836737    2582 scope.go:117] "RemoveContainer" containerID="114aaf91e84c4953b5ebdc54f20f5bab01dfac819f51f2230e59b3c39621d06f"
Sep 17 18:31:19.857831 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.857797    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f5c89875-fdftv"]
Sep 17 18:31:19.861195 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:19.861171    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74f5c89875-fdftv"]
Sep 17 18:31:20.181790 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:20.181711    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5741ee57-541c-4e53-a5fa-164c75701636" path="/var/lib/kubelet/pods/5741ee57-541c-4e53-a5fa-164c75701636/volumes"
Sep 17 18:31:28.622851 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:28.622817    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:31:28.629377 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:31:28.629339    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8f958cdc9-clxc2"
Sep 17 18:32:04.017297 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:04.017261    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:32:04.017844 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:04.017597    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy-metric" containerID="cri-o://016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e" gracePeriod=120
Sep 17 18:32:04.017844 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:04.017623    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy-web" containerID="cri-o://7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743" gracePeriod=120
Sep 17 18:32:04.017844 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:04.017655    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="config-reloader" containerID="cri-o://2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956" gracePeriod=120
Sep 17 18:32:04.017844 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:04.017580    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="alertmanager" containerID="cri-o://51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e" gracePeriod=120
Sep 17 18:32:04.017844 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:04.017659    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy" containerID="cri-o://d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00" gracePeriod=120
Sep 17 18:32:04.017844 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:04.017601    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="prom-label-proxy" containerID="cri-o://de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4" gracePeriod=120
Sep 17 18:32:05.012955 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.012921    2582 generic.go:356] "Generic (PLEG): container finished" podID="03e99b64-9872-422f-af14-319e046351fd" containerID="de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4" exitCode=0
Sep 17 18:32:05.012955 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.012944    2582 generic.go:356] "Generic (PLEG): container finished" podID="03e99b64-9872-422f-af14-319e046351fd" containerID="016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e" exitCode=0
Sep 17 18:32:05.012955 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.012950    2582 generic.go:356] "Generic (PLEG): container finished" podID="03e99b64-9872-422f-af14-319e046351fd" containerID="d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00" exitCode=0
Sep 17 18:32:05.012955 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.012956    2582 generic.go:356] "Generic (PLEG): container finished" podID="03e99b64-9872-422f-af14-319e046351fd" containerID="2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956" exitCode=0
Sep 17 18:32:05.012955 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.012961    2582 generic.go:356] "Generic (PLEG): container finished" podID="03e99b64-9872-422f-af14-319e046351fd" containerID="51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e" exitCode=0
Sep 17 18:32:05.013256 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.013000    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerDied","Data":"de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4"}
Sep 17 18:32:05.013256 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.013064    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerDied","Data":"016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e"}
Sep 17 18:32:05.013256 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.013091    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerDied","Data":"d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00"}
Sep 17 18:32:05.013256 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.013115    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerDied","Data":"2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956"}
Sep 17 18:32:05.013256 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.013137    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerDied","Data":"51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e"}
Sep 17 18:32:05.259799 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.259767    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:05.339534 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339505    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-alertmanager-main-db\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.339733 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339547    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-metrics-client-ca\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.339733 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339633    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.339733 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339665    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-web\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.339733 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339694    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-767jb\" (UniqueName: \"kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-kube-api-access-767jb\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.339733 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339720    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-tls-assets\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.340009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339745    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-config-volume\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.340009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339786    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-config-out\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.340009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339817    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-main-tls\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.340009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339845    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.340009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339870    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:05.340009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339915    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-alertmanager-trusted-ca-bundle\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.340009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339945    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-cluster-tls-config\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.340009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.339973    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-web-config\") pod \"03e99b64-9872-422f-af14-319e046351fd\" (UID: \"03e99b64-9872-422f-af14-319e046351fd\") "
Sep 17 18:32:05.340362 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.340023    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:05.340362 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.340207    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-alertmanager-main-db\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.340362 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.340225    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-metrics-client-ca\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.340525 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.340404    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:05.343005 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.342978    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-config-out" (OuterVolumeSpecName: "config-out") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:05.343722 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.343680    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:05.343852 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.343713    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.344464 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.344434    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-config-volume" (OuterVolumeSpecName: "config-volume") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.344595 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.344533    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-kube-api-access-767jb" (OuterVolumeSpecName: "kube-api-access-767jb") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "kube-api-access-767jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:05.345116 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.345061    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.345388 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.345355    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.345516 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.345421    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.348177 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.348143    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.355536 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.355500    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-web-config" (OuterVolumeSpecName: "web-config") pod "03e99b64-9872-422f-af14-319e046351fd" (UID: "03e99b64-9872-422f-af14-319e046351fd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:05.440777 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.440733    2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03e99b64-9872-422f-af14-319e046351fd-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.440777 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.440770    2582 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-cluster-tls-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.440777 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.440783    2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-web-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.441078 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.440795    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.441078 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.440808    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.441078 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.440820    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-767jb\" (UniqueName: \"kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-kube-api-access-767jb\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.441078 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.440830    2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03e99b64-9872-422f-af14-319e046351fd-tls-assets\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.441078 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.440840    2582 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-config-volume\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.441078 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.440850    2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03e99b64-9872-422f-af14-319e046351fd-config-out\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.441078 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.440861    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-main-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:05.441078 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:05.440873    2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03e99b64-9872-422f-af14-319e046351fd-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:32:06.019945 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.019845    2582 generic.go:356] "Generic (PLEG): container finished" podID="03e99b64-9872-422f-af14-319e046351fd" containerID="7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743" exitCode=0
Sep 17 18:32:06.020084 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.019936    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerDied","Data":"7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743"}
Sep 17 18:32:06.020084 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.019959    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.020084 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.020042    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03e99b64-9872-422f-af14-319e046351fd","Type":"ContainerDied","Data":"dd2fea4d79c6fdf62390c4a53bc8fea85eed4138f4e3d7a9096eb9358395a7fe"}
Sep 17 18:32:06.020084 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.020076    2582 scope.go:117] "RemoveContainer" containerID="de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4"
Sep 17 18:32:06.028052 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.028035    2582 scope.go:117] "RemoveContainer" containerID="016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e"
Sep 17 18:32:06.035729 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.035710    2582 scope.go:117] "RemoveContainer" containerID="d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00"
Sep 17 18:32:06.043138 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.043115    2582 scope.go:117] "RemoveContainer" containerID="7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743"
Sep 17 18:32:06.044609 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.044587    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:32:06.050962 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.050935    2582 scope.go:117] "RemoveContainer" containerID="2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956"
Sep 17 18:32:06.052060 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.052047    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:32:06.058794 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.058775    2582 scope.go:117] "RemoveContainer" containerID="51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e"
Sep 17 18:32:06.066682 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.066664    2582 scope.go:117] "RemoveContainer" containerID="4e9cf9ee90815b731825e0d100df21cbf4e07219049a07cfb4e0f9f81790bddc"
Sep 17 18:32:06.074158 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.074138    2582 scope.go:117] "RemoveContainer" containerID="de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4"
Sep 17 18:32:06.074471 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:32:06.074449    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4\": container with ID starting with de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4 not found: ID does not exist" containerID="de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4"
Sep 17 18:32:06.074518 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.074481    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4"} err="failed to get container status \"de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4\": rpc error: code = NotFound desc = could not find container \"de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4\": container with ID starting with de0017d548d25d7ae451ad74890d57c04605b976805e055dd6537d5fce462cf4 not found: ID does not exist"
Sep 17 18:32:06.074518 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.074514    2582 scope.go:117] "RemoveContainer" containerID="016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e"
Sep 17 18:32:06.074792 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:32:06.074773    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e\": container with ID starting with 016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e not found: ID does not exist" containerID="016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e"
Sep 17 18:32:06.074830 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.074798    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e"} err="failed to get container status \"016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e\": rpc error: code = NotFound desc = could not find container \"016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e\": container with ID starting with 016a10ac8c19ac523cc94399ed14d53474d841db51f14f3a1fbc77028f186b7e not found: ID does not exist"
Sep 17 18:32:06.074830 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.074814    2582 scope.go:117] "RemoveContainer" containerID="d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00"
Sep 17 18:32:06.075063 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:32:06.075046    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00\": container with ID starting with d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00 not found: ID does not exist" containerID="d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00"
Sep 17 18:32:06.075114 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.075069    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00"} err="failed to get container status \"d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00\": rpc error: code = NotFound desc = could not find container \"d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00\": container with ID starting with d51fe40025021cc449c8a5ad86ad47b1e79f2a2f53060ff03993ea82215a6b00 not found: ID does not exist"
Sep 17 18:32:06.075114 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.075084    2582 scope.go:117] "RemoveContainer" containerID="7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743"
Sep 17 18:32:06.075325 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:32:06.075308    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743\": container with ID starting with 7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743 not found: ID does not exist" containerID="7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743"
Sep 17 18:32:06.075368 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.075331    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743"} err="failed to get container status \"7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743\": rpc error: code = NotFound desc = could not find container \"7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743\": container with ID starting with 7f4df222737d270a3829fee394dff28ee2ebd93c83afd3640d0c6020e9e86743 not found: ID does not exist"
Sep 17 18:32:06.075368 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.075347    2582 scope.go:117] "RemoveContainer" containerID="2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956"
Sep 17 18:32:06.075566 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:32:06.075550    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956\": container with ID starting with 2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956 not found: ID does not exist" containerID="2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956"
Sep 17 18:32:06.075607 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.075572    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956"} err="failed to get container status \"2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956\": rpc error: code = NotFound desc = could not find container \"2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956\": container with ID starting with 2a89b7cef44d30ffef3c5bc6766c89a7e42c27e02b4d201a76fd3a08e898b956 not found: ID does not exist"
Sep 17 18:32:06.075607 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.075586    2582 scope.go:117] "RemoveContainer" containerID="51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e"
Sep 17 18:32:06.075816 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:32:06.075800    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e\": container with ID starting with 51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e not found: ID does not exist" containerID="51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e"
Sep 17 18:32:06.075858 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.075820    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e"} err="failed to get container status \"51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e\": rpc error: code = NotFound desc = could not find container \"51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e\": container with ID starting with 51487bc532c9d1ceddf04fb2e75593d5de65f72efaed0fd4ca06da5eaa40120e not found: ID does not exist"
Sep 17 18:32:06.075858 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.075835    2582 scope.go:117] "RemoveContainer" containerID="4e9cf9ee90815b731825e0d100df21cbf4e07219049a07cfb4e0f9f81790bddc"
Sep 17 18:32:06.076093 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:32:06.076076    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9cf9ee90815b731825e0d100df21cbf4e07219049a07cfb4e0f9f81790bddc\": container with ID starting with 4e9cf9ee90815b731825e0d100df21cbf4e07219049a07cfb4e0f9f81790bddc not found: ID does not exist" containerID="4e9cf9ee90815b731825e0d100df21cbf4e07219049a07cfb4e0f9f81790bddc"
Sep 17 18:32:06.076137 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.076098    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9cf9ee90815b731825e0d100df21cbf4e07219049a07cfb4e0f9f81790bddc"} err="failed to get container status \"4e9cf9ee90815b731825e0d100df21cbf4e07219049a07cfb4e0f9f81790bddc\": rpc error: code = NotFound desc = could not find container \"4e9cf9ee90815b731825e0d100df21cbf4e07219049a07cfb4e0f9f81790bddc\": container with ID starting with 4e9cf9ee90815b731825e0d100df21cbf4e07219049a07cfb4e0f9f81790bddc not found: ID does not exist"
Sep 17 18:32:06.082275 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082249    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:32:06.082566 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082550    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5741ee57-541c-4e53-a5fa-164c75701636" containerName="console"
Sep 17 18:32:06.082645 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082569    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5741ee57-541c-4e53-a5fa-164c75701636" containerName="console"
Sep 17 18:32:06.082645 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082583    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="config-reloader"
Sep 17 18:32:06.082645 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082593    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="config-reloader"
Sep 17 18:32:06.082645 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082609    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="alertmanager"
Sep 17 18:32:06.082645 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082617    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="alertmanager"
Sep 17 18:32:06.082645 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082627    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy-metric"
Sep 17 18:32:06.082645 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082635    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy-metric"
Sep 17 18:32:06.082645 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082645    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="prom-label-proxy"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082653    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="prom-label-proxy"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082665    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy-web"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082674    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy-web"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082691    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082699    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082707    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33bc31b2-3547-4014-b2f8-b621f05d729e" containerName="registry"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082715    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33bc31b2-3547-4014-b2f8-b621f05d729e" containerName="registry"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082726    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="init-config-reloader"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082734    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="init-config-reloader"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082801    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="alertmanager"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082812    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy-metric"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082821    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082828    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="prom-label-proxy"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082837    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="config-reloader"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082845    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e99b64-9872-422f-af14-319e046351fd" containerName="kube-rbac-proxy-web"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082854    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5741ee57-541c-4e53-a5fa-164c75701636" containerName="console"
Sep 17 18:32:06.083043 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.082862    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33bc31b2-3547-4014-b2f8-b621f05d729e" containerName="registry"
Sep 17 18:32:06.086862 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.086839    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.089482 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.089460    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 18:32:06.089810 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.089790    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 18:32:06.090096 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.090078    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-x7kzt\""
Sep 17 18:32:06.090185 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.090100    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 18:32:06.090185 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.090087    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 18:32:06.090185 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.090178    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 18:32:06.090458 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.090443    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 18:32:06.091016 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.091000    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 18:32:06.091426 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.091407    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 18:32:06.099520 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.099490    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 18:32:06.107799 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.107767    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:32:06.146482 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146439    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.146482 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146482    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.146720 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146505    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-web-config\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.146720 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146529    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.146720 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-config-volume\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.146720 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.146720 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146661    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzp4c\" (UniqueName: \"kubernetes.io/projected/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-kube-api-access-tzp4c\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.146720 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.146720 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146701    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.146720 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-config-out\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.147013 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146779    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.147013 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146809    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.147013 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.146833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.181391 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.181353    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03e99b64-9872-422f-af14-319e046351fd" path="/var/lib/kubelet/pods/03e99b64-9872-422f-af14-319e046351fd/volumes"
Sep 17 18:32:06.248058 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-web-config\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-config-volume\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248517 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248517 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzp4c\" (UniqueName: \"kubernetes.io/projected/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-kube-api-access-tzp4c\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248517 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248369    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248517 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248517 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-config-out\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248517 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.248517 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.249023 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.248876    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.249130 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.249030    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.249686 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.249654    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.251876 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.251745    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.252201 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.252171    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-config-volume\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.252330 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.252310    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.252398 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.252381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-web-config\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.252450 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.252427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.252450 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.252437    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.252550 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.252485    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.252550 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.252498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-config-out\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.253240 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.253223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.257582 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.257555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzp4c\" (UniqueName: \"kubernetes.io/projected/6377d1a3-33b0-4c59-8a8e-1a7b24e639da-kube-api-access-tzp4c\") pod \"alertmanager-main-0\" (UID: \"6377d1a3-33b0-4c59-8a8e-1a7b24e639da\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.396692 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.396650    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:32:06.681778 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:06.681694    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:32:06.687258 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:32:06.687231    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6377d1a3_33b0_4c59_8a8e_1a7b24e639da.slice/crio-20382a485c1462095e1f796743cd2590c8af9eaa21c0f920eee19c9897e69e91 WatchSource:0}: Error finding container 20382a485c1462095e1f796743cd2590c8af9eaa21c0f920eee19c9897e69e91: Status 404 returned error can't find the container with id 20382a485c1462095e1f796743cd2590c8af9eaa21c0f920eee19c9897e69e91
Sep 17 18:32:07.026538 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:07.026449    2582 generic.go:356] "Generic (PLEG): container finished" podID="6377d1a3-33b0-4c59-8a8e-1a7b24e639da" containerID="14d6474bd72e5e9d5962562e2494225a4266344aab391ed4a64987e01d44c23c" exitCode=0
Sep 17 18:32:07.026538 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:07.026495    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6377d1a3-33b0-4c59-8a8e-1a7b24e639da","Type":"ContainerDied","Data":"14d6474bd72e5e9d5962562e2494225a4266344aab391ed4a64987e01d44c23c"}
Sep 17 18:32:07.026711 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:07.026556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6377d1a3-33b0-4c59-8a8e-1a7b24e639da","Type":"ContainerStarted","Data":"20382a485c1462095e1f796743cd2590c8af9eaa21c0f920eee19c9897e69e91"}
Sep 17 18:32:08.035230 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.035189    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6377d1a3-33b0-4c59-8a8e-1a7b24e639da","Type":"ContainerStarted","Data":"7ce88e3457c190c6b8cd7df39c9f0b720022a79ee77525107c0b0f0d1fe769a7"}
Sep 17 18:32:08.035688 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.035307    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6377d1a3-33b0-4c59-8a8e-1a7b24e639da","Type":"ContainerStarted","Data":"60ada5bbadd80eccc0f92b7e447f3c100eebe6bbb3867d477fa34449fc66c501"}
Sep 17 18:32:08.035688 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.035352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6377d1a3-33b0-4c59-8a8e-1a7b24e639da","Type":"ContainerStarted","Data":"856413431efed7123dfac56bbed5dbde085459bcd30f2095da1aa1b223553eba"}
Sep 17 18:32:08.035688 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.035402    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6377d1a3-33b0-4c59-8a8e-1a7b24e639da","Type":"ContainerStarted","Data":"567e85f5c0748610640015968295f544e6aad04193db7574f93a0008078ae2ce"}
Sep 17 18:32:08.035688 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.035440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6377d1a3-33b0-4c59-8a8e-1a7b24e639da","Type":"ContainerStarted","Data":"54730c720fa7add5be49a593fda9bac807a352bda7e2b18800f369083208bea9"}
Sep 17 18:32:08.035688 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.035480    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6377d1a3-33b0-4c59-8a8e-1a7b24e639da","Type":"ContainerStarted","Data":"1305b4b8e9ece29285ee1396c8792db3b44367dcf27b0f4c24071c68a2cff522"}
Sep 17 18:32:08.065874 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.065842    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"]
Sep 17 18:32:08.068358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.068330    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.073074 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.073046    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 18:32:08.073451 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.073431    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 18:32:08.073558 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.073538    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 18:32:08.074329 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.074308    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6w6hs\""
Sep 17 18:32:08.075133 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.075109    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 18:32:08.075317 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.075278    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.059078885 podStartE2EDuration="2.075262927s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.184042ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:08.073650684 +0000 UTC m=+161.997433346" watchObservedRunningTime="2026-09-17 18:32:08.075262927 +0000 UTC m=+161.999045585"
Sep 17 18:32:08.077553 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.077533    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 18:32:08.081062 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.081044    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 18:32:08.096249 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.096171    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"]
Sep 17 18:32:08.165701 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.165630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.165701 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.165665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab102fd-f9dc-4955-9a6f-8b422689c64c-metrics-client-ca\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.165701 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.165694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ab102fd-f9dc-4955-9a6f-8b422689c64c-serving-certs-ca-bundle\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.165924 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.165738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ab102fd-f9dc-4955-9a6f-8b422689c64c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.165924 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.165838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-federate-client-tls\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.165987 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.165928    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-secret-telemeter-client\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.166023 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.165984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rl4x\" (UniqueName: \"kubernetes.io/projected/5ab102fd-f9dc-4955-9a6f-8b422689c64c-kube-api-access-7rl4x\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.166056 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.166020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-telemeter-client-tls\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.267323 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.267281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-federate-client-tls\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.267512 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.267335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-secret-telemeter-client\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.267512 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.267390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rl4x\" (UniqueName: \"kubernetes.io/projected/5ab102fd-f9dc-4955-9a6f-8b422689c64c-kube-api-access-7rl4x\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.267512 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.267427    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-telemeter-client-tls\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.267512 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.267452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.267512 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.267481    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab102fd-f9dc-4955-9a6f-8b422689c64c-metrics-client-ca\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.267512 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.267510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ab102fd-f9dc-4955-9a6f-8b422689c64c-serving-certs-ca-bundle\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.267814 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.267531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ab102fd-f9dc-4955-9a6f-8b422689c64c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.268623 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.268585    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ab102fd-f9dc-4955-9a6f-8b422689c64c-metrics-client-ca\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.268736 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.268638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ab102fd-f9dc-4955-9a6f-8b422689c64c-serving-certs-ca-bundle\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.268852 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.268828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ab102fd-f9dc-4955-9a6f-8b422689c64c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.270240 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.270218    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-federate-client-tls\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.270363 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.270346    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.270415 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.270397    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-secret-telemeter-client\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.270992 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.270975    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5ab102fd-f9dc-4955-9a6f-8b422689c64c-telemeter-client-tls\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.280530 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.280505    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rl4x\" (UniqueName: \"kubernetes.io/projected/5ab102fd-f9dc-4955-9a6f-8b422689c64c-kube-api-access-7rl4x\") pod \"telemeter-client-6fd699cdf4-sqn24\" (UID: \"5ab102fd-f9dc-4955-9a6f-8b422689c64c\") " pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.380800 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.380756    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"
Sep 17 18:32:08.678573 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:08.678532    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6fd699cdf4-sqn24"]
Sep 17 18:32:08.681456 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:32:08.681425    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab102fd_f9dc_4955_9a6f_8b422689c64c.slice/crio-bf00ebe54c70014ca42eb379ad2fabb926e48dae704c5c48647fc0f258f01ff8 WatchSource:0}: Error finding container bf00ebe54c70014ca42eb379ad2fabb926e48dae704c5c48647fc0f258f01ff8: Status 404 returned error can't find the container with id bf00ebe54c70014ca42eb379ad2fabb926e48dae704c5c48647fc0f258f01ff8
Sep 17 18:32:09.040192 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:09.040097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24" event={"ID":"5ab102fd-f9dc-4955-9a6f-8b422689c64c","Type":"ContainerStarted","Data":"bf00ebe54c70014ca42eb379ad2fabb926e48dae704c5c48647fc0f258f01ff8"}
Sep 17 18:32:11.051395 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.051357    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24" event={"ID":"5ab102fd-f9dc-4955-9a6f-8b422689c64c","Type":"ContainerStarted","Data":"478388537b8fa1b64e8284a0423a4e9490dabb4d6048728cc355500a9b6fc0db"}
Sep 17 18:32:11.051832 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.051418    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24" event={"ID":"5ab102fd-f9dc-4955-9a6f-8b422689c64c","Type":"ContainerStarted","Data":"4dd13e77fd00c7ecef90000d04609e09c0b7c5eb8e39512fe1afce9ea4e620a6"}
Sep 17 18:32:11.051832 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.051438    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24" event={"ID":"5ab102fd-f9dc-4955-9a6f-8b422689c64c","Type":"ContainerStarted","Data":"05541d50bcc2ebbf66edc5c7dfbf658a663b00d5c1d03c5652ebf3055b3700d0"}
Sep 17 18:32:11.081211 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.081154    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6fd699cdf4-sqn24" podStartSLOduration=1.496497198 podStartE2EDuration="3.081135695s" totalImagesPullingTime="1.584638497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:11.080815886 +0000 UTC m=+165.004598548" watchObservedRunningTime="2026-09-17 18:32:11.081135695 +0000 UTC m=+165.004918356"
Sep 17 18:32:11.187078 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.187049    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67568cddb-vxrxx"]
Sep 17 18:32:11.190224 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.190207    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.192365 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.192344    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 18:32:11.192942 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.192922    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 18:32:11.193058 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.192928    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 18:32:11.193298 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.193281    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hdv42\""
Sep 17 18:32:11.193379 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.193354    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 18:32:11.193434 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.193397    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 18:32:11.198763 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.198741    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 18:32:11.207477 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.207454    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67568cddb-vxrxx"]
Sep 17 18:32:11.296453 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.296412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-service-ca\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.296666 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.296481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-oauth-config\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.296666 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.296528    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-trusted-ca-bundle\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.296666 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.296575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-oauth-serving-cert\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.296666 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.296602    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg6xp\" (UniqueName: \"kubernetes.io/projected/d9d8c69b-34e8-4058-9f87-5db29247cdb2-kube-api-access-tg6xp\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.296666 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.296644    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-config\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.296973 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.296687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-serving-cert\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.398254 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.398161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-config\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.398254 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.398205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-serving-cert\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.398254 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.398255    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-service-ca\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.398537 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.398285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-oauth-config\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.398537 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.398305    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-trusted-ca-bundle\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.398537 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.398334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-oauth-serving-cert\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.398537 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.398357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg6xp\" (UniqueName: \"kubernetes.io/projected/d9d8c69b-34e8-4058-9f87-5db29247cdb2-kube-api-access-tg6xp\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.399134 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.399103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-oauth-serving-cert\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.399242 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.399128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-config\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.399313 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.399293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-service-ca\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.401033 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.399852    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-trusted-ca-bundle\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.401477 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.401453    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-oauth-config\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.406024 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.405990    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-serving-cert\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.408373 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.408347    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg6xp\" (UniqueName: \"kubernetes.io/projected/d9d8c69b-34e8-4058-9f87-5db29247cdb2-kube-api-access-tg6xp\") pod \"console-67568cddb-vxrxx\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") " pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.501363 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.501325    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:11.778222 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:11.778149    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67568cddb-vxrxx"]
Sep 17 18:32:11.781398 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:32:11.781369    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d8c69b_34e8_4058_9f87_5db29247cdb2.slice/crio-29bdbce3bbe15edf6b30737c1c2e24e4d5167fb46f403115751d5a2ab6cb94d6 WatchSource:0}: Error finding container 29bdbce3bbe15edf6b30737c1c2e24e4d5167fb46f403115751d5a2ab6cb94d6: Status 404 returned error can't find the container with id 29bdbce3bbe15edf6b30737c1c2e24e4d5167fb46f403115751d5a2ab6cb94d6
Sep 17 18:32:12.057858 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:12.057812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67568cddb-vxrxx" event={"ID":"d9d8c69b-34e8-4058-9f87-5db29247cdb2","Type":"ContainerStarted","Data":"d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9"}
Sep 17 18:32:12.058288 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:12.057877    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67568cddb-vxrxx" event={"ID":"d9d8c69b-34e8-4058-9f87-5db29247cdb2","Type":"ContainerStarted","Data":"29bdbce3bbe15edf6b30737c1c2e24e4d5167fb46f403115751d5a2ab6cb94d6"}
Sep 17 18:32:21.501659 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:21.501609    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:21.502099 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:21.501682    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:21.506454 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:21.506426    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:32:21.542634 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:21.542573    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67568cddb-vxrxx" podStartSLOduration=10.542555501 podStartE2EDuration="10.542555501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:12.076536387 +0000 UTC m=+166.000319052" watchObservedRunningTime="2026-09-17 18:32:21.542555501 +0000 UTC m=+175.466338162"
Sep 17 18:32:22.100004 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:32:22.099976    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:33:20.023380 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.023344    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bc8cd6744-9w88f"]
Sep 17 18:33:20.026866 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.026847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.045708 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.045676    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc8cd6744-9w88f"]
Sep 17 18:33:20.061094 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.061059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-trusted-ca-bundle\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.061250 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.061113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-service-ca\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.061250 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.061164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-serving-cert\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.061250 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.061188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-config\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.061368 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.061249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-oauth-config\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.061368 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.061300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-oauth-serving-cert\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.061368 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.061317    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m8rp\" (UniqueName: \"kubernetes.io/projected/6a5a219b-4f85-4188-bb52-7ba81ab89896-kube-api-access-9m8rp\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.161943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.161877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-oauth-serving-cert\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.161943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.161941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m8rp\" (UniqueName: \"kubernetes.io/projected/6a5a219b-4f85-4188-bb52-7ba81ab89896-kube-api-access-9m8rp\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.162201 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.161966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-trusted-ca-bundle\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.162201 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.162001    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-service-ca\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.162303 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.162221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-serving-cert\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.162303 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.162272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-config\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.162406 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.162314    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-oauth-config\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.162612 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.162588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-oauth-serving-cert\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.162796 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.162768    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-service-ca\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.162864 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.162810    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-trusted-ca-bundle\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.162959 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.162857    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-config\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.164966 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.164942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-oauth-config\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.165066 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.164972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-serving-cert\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.178141 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.178114    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m8rp\" (UniqueName: \"kubernetes.io/projected/6a5a219b-4f85-4188-bb52-7ba81ab89896-kube-api-access-9m8rp\") pod \"console-5bc8cd6744-9w88f\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") " pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.336523 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.336483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:20.614785 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:20.614626    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc8cd6744-9w88f"]
Sep 17 18:33:20.616963 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:33:20.616924    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5a219b_4f85_4188_bb52_7ba81ab89896.slice/crio-45a9cead9f6d5f2c96ed7a222e9b36b4e53cd637c51e2034948c2d3ae465ec96 WatchSource:0}: Error finding container 45a9cead9f6d5f2c96ed7a222e9b36b4e53cd637c51e2034948c2d3ae465ec96: Status 404 returned error can't find the container with id 45a9cead9f6d5f2c96ed7a222e9b36b4e53cd637c51e2034948c2d3ae465ec96
Sep 17 18:33:21.320181 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:21.320141    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc8cd6744-9w88f" event={"ID":"6a5a219b-4f85-4188-bb52-7ba81ab89896","Type":"ContainerStarted","Data":"4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b"}
Sep 17 18:33:21.320609 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:21.320195    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc8cd6744-9w88f" event={"ID":"6a5a219b-4f85-4188-bb52-7ba81ab89896","Type":"ContainerStarted","Data":"45a9cead9f6d5f2c96ed7a222e9b36b4e53cd637c51e2034948c2d3ae465ec96"}
Sep 17 18:33:21.345352 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:21.345299    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bc8cd6744-9w88f" podStartSLOduration=2.345281314 podStartE2EDuration="2.345281314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:21.343056703 +0000 UTC m=+235.266839377" watchObservedRunningTime="2026-09-17 18:33:21.345281314 +0000 UTC m=+235.269063975"
Sep 17 18:33:30.336767 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:30.336719    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:30.337220 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:30.336790    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:30.341559 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:30.341538    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:30.359600 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:30.359571    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:33:30.425831 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:30.425798    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67568cddb-vxrxx"]
Sep 17 18:33:55.445836 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.445730    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67568cddb-vxrxx" podUID="d9d8c69b-34e8-4058-9f87-5db29247cdb2" containerName="console" containerID="cri-o://d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9" gracePeriod=15
Sep 17 18:33:55.687503 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.687478    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:33:55.771784 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.771702    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-trusted-ca-bundle\") pod \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") "
Sep 17 18:33:55.771784 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.771743    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-serving-cert\") pod \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") "
Sep 17 18:33:55.771784 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.771763    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-config\") pod \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") "
Sep 17 18:33:55.772071 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.771795    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-oauth-serving-cert\") pod \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") "
Sep 17 18:33:55.772071 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.771858    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-service-ca\") pod \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") "
Sep 17 18:33:55.772071 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.771916    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg6xp\" (UniqueName: \"kubernetes.io/projected/d9d8c69b-34e8-4058-9f87-5db29247cdb2-kube-api-access-tg6xp\") pod \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") "
Sep 17 18:33:55.772071 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.771944    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-oauth-config\") pod \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\" (UID: \"d9d8c69b-34e8-4058-9f87-5db29247cdb2\") "
Sep 17 18:33:55.772278 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.772244    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d9d8c69b-34e8-4058-9f87-5db29247cdb2" (UID: "d9d8c69b-34e8-4058-9f87-5db29247cdb2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:55.772278 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.772226    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-config" (OuterVolumeSpecName: "console-config") pod "d9d8c69b-34e8-4058-9f87-5db29247cdb2" (UID: "d9d8c69b-34e8-4058-9f87-5db29247cdb2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:55.772387 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.772321    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d9d8c69b-34e8-4058-9f87-5db29247cdb2" (UID: "d9d8c69b-34e8-4058-9f87-5db29247cdb2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:55.772664 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.772643    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-service-ca" (OuterVolumeSpecName: "service-ca") pod "d9d8c69b-34e8-4058-9f87-5db29247cdb2" (UID: "d9d8c69b-34e8-4058-9f87-5db29247cdb2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:55.774319 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.774285    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d8c69b-34e8-4058-9f87-5db29247cdb2-kube-api-access-tg6xp" (OuterVolumeSpecName: "kube-api-access-tg6xp") pod "d9d8c69b-34e8-4058-9f87-5db29247cdb2" (UID: "d9d8c69b-34e8-4058-9f87-5db29247cdb2"). InnerVolumeSpecName "kube-api-access-tg6xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:55.774436 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.774294    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d9d8c69b-34e8-4058-9f87-5db29247cdb2" (UID: "d9d8c69b-34e8-4058-9f87-5db29247cdb2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:55.774436 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.774390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d9d8c69b-34e8-4058-9f87-5db29247cdb2" (UID: "d9d8c69b-34e8-4058-9f87-5db29247cdb2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:55.872588 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.872552    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-service-ca\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:33:55.872588 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.872580    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg6xp\" (UniqueName: \"kubernetes.io/projected/d9d8c69b-34e8-4058-9f87-5db29247cdb2-kube-api-access-tg6xp\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:33:55.872588 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.872594    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-oauth-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:33:55.872821 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.872603    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-trusted-ca-bundle\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:33:55.872821 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.872613    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-serving-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:33:55.872821 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.872622    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-console-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:33:55.872821 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:55.872631    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9d8c69b-34e8-4058-9f87-5db29247cdb2-oauth-serving-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:33:56.463742 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:56.463652    2582 generic.go:356] "Generic (PLEG): container finished" podID="d9d8c69b-34e8-4058-9f87-5db29247cdb2" containerID="d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9" exitCode=0
Sep 17 18:33:56.463742 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:56.463701    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67568cddb-vxrxx"
Sep 17 18:33:56.464183 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:56.463736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67568cddb-vxrxx" event={"ID":"d9d8c69b-34e8-4058-9f87-5db29247cdb2","Type":"ContainerDied","Data":"d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9"}
Sep 17 18:33:56.464183 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:56.463791    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67568cddb-vxrxx" event={"ID":"d9d8c69b-34e8-4058-9f87-5db29247cdb2","Type":"ContainerDied","Data":"29bdbce3bbe15edf6b30737c1c2e24e4d5167fb46f403115751d5a2ab6cb94d6"}
Sep 17 18:33:56.464183 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:56.463812    2582 scope.go:117] "RemoveContainer" containerID="d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9"
Sep 17 18:33:56.472078 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:56.472058    2582 scope.go:117] "RemoveContainer" containerID="d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9"
Sep 17 18:33:56.472360 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:33:56.472341    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9\": container with ID starting with d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9 not found: ID does not exist" containerID="d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9"
Sep 17 18:33:56.472411 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:56.472370    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9"} err="failed to get container status \"d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9\": rpc error: code = NotFound desc = could not find container \"d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9\": container with ID starting with d063561efb96436d285740528ad0e2dd3a206e32a737fbedb8d135a5ceaa0da9 not found: ID does not exist"
Sep 17 18:33:56.491741 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:56.491711    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67568cddb-vxrxx"]
Sep 17 18:33:56.497558 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:56.497532    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67568cddb-vxrxx"]
Sep 17 18:33:58.182175 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:33:58.182138    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9d8c69b-34e8-4058-9f87-5db29247cdb2" path="/var/lib/kubelet/pods/d9d8c69b-34e8-4058-9f87-5db29247cdb2/volumes"
Sep 17 18:34:28.126754 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:28.126730    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:34:37.095656 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.095619    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"]
Sep 17 18:34:37.098044 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.095934    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d8c69b-34e8-4058-9f87-5db29247cdb2" containerName="console"
Sep 17 18:34:37.098044 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.095946    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d8c69b-34e8-4058-9f87-5db29247cdb2" containerName="console"
Sep 17 18:34:37.098044 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.096003    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d8c69b-34e8-4058-9f87-5db29247cdb2" containerName="console"
Sep 17 18:34:37.098917 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.098901    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:37.101494 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.101469    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 18:34:37.102049 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.102031    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-9bxld\""
Sep 17 18:34:37.102320 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.102299    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 18:34:37.124258 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.124225    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"]
Sep 17 18:34:37.193130 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.193094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:37.193305 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.193162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:37.193305 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.193224    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkj2b\" (UniqueName: \"kubernetes.io/projected/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-kube-api-access-dkj2b\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:37.294375 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.294331    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:37.294588 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.294398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkj2b\" (UniqueName: \"kubernetes.io/projected/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-kube-api-access-dkj2b\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:37.294588 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.294442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:37.294795 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.294769    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:37.294844 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.294812    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:37.305100 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.305076    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkj2b\" (UniqueName: \"kubernetes.io/projected/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-kube-api-access-dkj2b\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:37.408470 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.408383    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:37.685010 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.684987    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"]
Sep 17 18:34:37.687606 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:34:37.687573    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52b9971_7f3b_4c00_a380_2aa9b2dd950b.slice/crio-da25769a5403d92b663716e93e1e3ef4b022a4b9e9fa7ac1d3ff2277fff05b69 WatchSource:0}: Error finding container da25769a5403d92b663716e93e1e3ef4b022a4b9e9fa7ac1d3ff2277fff05b69: Status 404 returned error can't find the container with id da25769a5403d92b663716e93e1e3ef4b022a4b9e9fa7ac1d3ff2277fff05b69
Sep 17 18:34:37.689675 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:37.689660    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:34:38.629859 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:38.629825    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5" event={"ID":"b52b9971-7f3b-4c00-a380-2aa9b2dd950b","Type":"ContainerStarted","Data":"da25769a5403d92b663716e93e1e3ef4b022a4b9e9fa7ac1d3ff2277fff05b69"}
Sep 17 18:34:44.657657 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:44.657621    2582 generic.go:356] "Generic (PLEG): container finished" podID="b52b9971-7f3b-4c00-a380-2aa9b2dd950b" containerID="0bcffcbffbf46198041b53fb97158ea840a7e4788b980283d3fb8cb73868c493" exitCode=0
Sep 17 18:34:44.658133 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:44.657706    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5" event={"ID":"b52b9971-7f3b-4c00-a380-2aa9b2dd950b","Type":"ContainerDied","Data":"0bcffcbffbf46198041b53fb97158ea840a7e4788b980283d3fb8cb73868c493"}
Sep 17 18:34:47.672416 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:47.672381    2582 generic.go:356] "Generic (PLEG): container finished" podID="b52b9971-7f3b-4c00-a380-2aa9b2dd950b" containerID="adcbb1965a815200e63dc6587c892e1065b3f6361d4d1b1dcb3c70137eff2f55" exitCode=0
Sep 17 18:34:47.672785 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:47.672440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5" event={"ID":"b52b9971-7f3b-4c00-a380-2aa9b2dd950b","Type":"ContainerDied","Data":"adcbb1965a815200e63dc6587c892e1065b3f6361d4d1b1dcb3c70137eff2f55"}
Sep 17 18:34:54.706404 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:54.706365    2582 generic.go:356] "Generic (PLEG): container finished" podID="b52b9971-7f3b-4c00-a380-2aa9b2dd950b" containerID="353207760a6f6a38b6e88644d2c98072526e85de3ff6cb5c88c6908f10d07266" exitCode=0
Sep 17 18:34:54.706856 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:54.706423    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5" event={"ID":"b52b9971-7f3b-4c00-a380-2aa9b2dd950b","Type":"ContainerDied","Data":"353207760a6f6a38b6e88644d2c98072526e85de3ff6cb5c88c6908f10d07266"}
Sep 17 18:34:55.838067 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:55.838040    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:34:55.961399 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:55.961309    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-bundle\") pod \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") "
Sep 17 18:34:55.961399 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:55.961377    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-util\") pod \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") "
Sep 17 18:34:55.961580 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:55.961407    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkj2b\" (UniqueName: \"kubernetes.io/projected/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-kube-api-access-dkj2b\") pod \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\" (UID: \"b52b9971-7f3b-4c00-a380-2aa9b2dd950b\") "
Sep 17 18:34:55.962008 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:55.961984    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-bundle" (OuterVolumeSpecName: "bundle") pod "b52b9971-7f3b-4c00-a380-2aa9b2dd950b" (UID: "b52b9971-7f3b-4c00-a380-2aa9b2dd950b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:55.963926 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:55.963865    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-kube-api-access-dkj2b" (OuterVolumeSpecName: "kube-api-access-dkj2b") pod "b52b9971-7f3b-4c00-a380-2aa9b2dd950b" (UID: "b52b9971-7f3b-4c00-a380-2aa9b2dd950b"). InnerVolumeSpecName "kube-api-access-dkj2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:34:55.966120 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:55.966092    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-util" (OuterVolumeSpecName: "util") pod "b52b9971-7f3b-4c00-a380-2aa9b2dd950b" (UID: "b52b9971-7f3b-4c00-a380-2aa9b2dd950b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:56.062596 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:56.062556    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-bundle\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:34:56.062596 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:56.062587    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-util\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:34:56.062596 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:56.062597    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkj2b\" (UniqueName: \"kubernetes.io/projected/b52b9971-7f3b-4c00-a380-2aa9b2dd950b-kube-api-access-dkj2b\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:34:56.715957 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:56.715926    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5" event={"ID":"b52b9971-7f3b-4c00-a380-2aa9b2dd950b","Type":"ContainerDied","Data":"da25769a5403d92b663716e93e1e3ef4b022a4b9e9fa7ac1d3ff2277fff05b69"}
Sep 17 18:34:56.716108 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:56.715972    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da25769a5403d92b663716e93e1e3ef4b022a4b9e9fa7ac1d3ff2277fff05b69"
Sep 17 18:34:56.716108 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:34:56.716016    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2qff5"
Sep 17 18:35:00.541390 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.541356    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"]
Sep 17 18:35:00.542009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.541822    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52b9971-7f3b-4c00-a380-2aa9b2dd950b" containerName="extract"
Sep 17 18:35:00.542009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.541841    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52b9971-7f3b-4c00-a380-2aa9b2dd950b" containerName="extract"
Sep 17 18:35:00.542009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.541858    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52b9971-7f3b-4c00-a380-2aa9b2dd950b" containerName="util"
Sep 17 18:35:00.542009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.541868    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52b9971-7f3b-4c00-a380-2aa9b2dd950b" containerName="util"
Sep 17 18:35:00.542009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.541916    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52b9971-7f3b-4c00-a380-2aa9b2dd950b" containerName="pull"
Sep 17 18:35:00.542009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.541925    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52b9971-7f3b-4c00-a380-2aa9b2dd950b" containerName="pull"
Sep 17 18:35:00.542009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.542013    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b52b9971-7f3b-4c00-a380-2aa9b2dd950b" containerName="extract"
Sep 17 18:35:00.596121 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.596092    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"
Sep 17 18:35:00.597973 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.597947    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"]
Sep 17 18:35:00.605793 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.605770    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 18:35:00.605960 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.605919    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-l5lg9\""
Sep 17 18:35:00.606329 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.606314    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 18:35:00.607744 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.607727    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 18:35:00.700157 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.700118    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4ctb\" (UniqueName: \"kubernetes.io/projected/adffef85-7a7f-42a8-8a00-9f072d8a941d-kube-api-access-v4ctb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwhv7\" (UID: \"adffef85-7a7f-42a8-8a00-9f072d8a941d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"
Sep 17 18:35:00.700332 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.700167    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/adffef85-7a7f-42a8-8a00-9f072d8a941d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwhv7\" (UID: \"adffef85-7a7f-42a8-8a00-9f072d8a941d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"
Sep 17 18:35:00.801570 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.801484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4ctb\" (UniqueName: \"kubernetes.io/projected/adffef85-7a7f-42a8-8a00-9f072d8a941d-kube-api-access-v4ctb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwhv7\" (UID: \"adffef85-7a7f-42a8-8a00-9f072d8a941d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"
Sep 17 18:35:00.801570 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.801538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/adffef85-7a7f-42a8-8a00-9f072d8a941d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwhv7\" (UID: \"adffef85-7a7f-42a8-8a00-9f072d8a941d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"
Sep 17 18:35:00.804032 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.804010    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/adffef85-7a7f-42a8-8a00-9f072d8a941d-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwhv7\" (UID: \"adffef85-7a7f-42a8-8a00-9f072d8a941d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"
Sep 17 18:35:00.810676 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.810649    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4ctb\" (UniqueName: \"kubernetes.io/projected/adffef85-7a7f-42a8-8a00-9f072d8a941d-kube-api-access-v4ctb\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-fwhv7\" (UID: \"adffef85-7a7f-42a8-8a00-9f072d8a941d\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"
Sep 17 18:35:00.906431 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:00.906387    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"
Sep 17 18:35:01.208559 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:01.208488    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"]
Sep 17 18:35:01.212362 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:35:01.212335    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadffef85_7a7f_42a8_8a00_9f072d8a941d.slice/crio-d7face08d0531d8d6da7cc4c52687d5840bc9db8ea1e036e2a06824c0d38d157 WatchSource:0}: Error finding container d7face08d0531d8d6da7cc4c52687d5840bc9db8ea1e036e2a06824c0d38d157: Status 404 returned error can't find the container with id d7face08d0531d8d6da7cc4c52687d5840bc9db8ea1e036e2a06824c0d38d157
Sep 17 18:35:01.740302 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:01.740269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7" event={"ID":"adffef85-7a7f-42a8-8a00-9f072d8a941d","Type":"ContainerStarted","Data":"d7face08d0531d8d6da7cc4c52687d5840bc9db8ea1e036e2a06824c0d38d157"}
Sep 17 18:35:05.762708 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.762657    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7" event={"ID":"adffef85-7a7f-42a8-8a00-9f072d8a941d","Type":"ContainerStarted","Data":"2bf547d172c3c5028ffbd0f43710e982944abdbd2eba1ff5031718c411e88e96"}
Sep 17 18:35:05.763157 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.762792    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"
Sep 17 18:35:05.786041 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.785993    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7" podStartSLOduration=1.9482776849999999 podStartE2EDuration="5.785977536s" totalImagesPullingTime="3.837699851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:05.783671397 +0000 UTC m=+339.707454081" watchObservedRunningTime="2026-09-17 18:35:05.785977536 +0000 UTC m=+339.709760197"
Sep 17 18:35:05.819262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.819226    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7l7z4"]
Sep 17 18:35:05.842156 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.842128    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7l7z4"]
Sep 17 18:35:05.842321 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.842228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:05.844856 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.844826    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 18:35:05.845018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.844925    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-mc7xb\""
Sep 17 18:35:05.845134 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.845117    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 18:35:05.952024 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.951979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de195ee8-17c8-401d-a050-5b63216be373-certificates\") pod \"keda-operator-579c8958f7-7l7z4\" (UID: \"de195ee8-17c8-401d-a050-5b63216be373\") " pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:05.952024 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.952026    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42g5d\" (UniqueName: \"kubernetes.io/projected/de195ee8-17c8-401d-a050-5b63216be373-kube-api-access-42g5d\") pod \"keda-operator-579c8958f7-7l7z4\" (UID: \"de195ee8-17c8-401d-a050-5b63216be373\") " pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:05.952275 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:05.952097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/de195ee8-17c8-401d-a050-5b63216be373-cabundle0\") pod \"keda-operator-579c8958f7-7l7z4\" (UID: \"de195ee8-17c8-401d-a050-5b63216be373\") " pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:06.053641 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.053593    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de195ee8-17c8-401d-a050-5b63216be373-certificates\") pod \"keda-operator-579c8958f7-7l7z4\" (UID: \"de195ee8-17c8-401d-a050-5b63216be373\") " pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:06.053641 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.053642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42g5d\" (UniqueName: \"kubernetes.io/projected/de195ee8-17c8-401d-a050-5b63216be373-kube-api-access-42g5d\") pod \"keda-operator-579c8958f7-7l7z4\" (UID: \"de195ee8-17c8-401d-a050-5b63216be373\") " pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:06.053914 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.053676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/de195ee8-17c8-401d-a050-5b63216be373-cabundle0\") pod \"keda-operator-579c8958f7-7l7z4\" (UID: \"de195ee8-17c8-401d-a050-5b63216be373\") " pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:06.053914 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:35:06.053777    2582 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 17 18:35:06.053914 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:35:06.053805    2582 secret.go:281] references non-existent secret key: ca.crt
Sep 17 18:35:06.053914 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:35:06.053817    2582 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 18:35:06.053914 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:35:06.053834    2582 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-7l7z4: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 18:35:06.054119 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:35:06.053923    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de195ee8-17c8-401d-a050-5b63216be373-certificates podName:de195ee8-17c8-401d-a050-5b63216be373 nodeName:}" failed. No retries permitted until 2026-09-17 18:35:06.553901486 +0000 UTC m=+340.477684140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/de195ee8-17c8-401d-a050-5b63216be373-certificates") pod "keda-operator-579c8958f7-7l7z4" (UID: "de195ee8-17c8-401d-a050-5b63216be373") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 18:35:06.054448 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.054428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/de195ee8-17c8-401d-a050-5b63216be373-cabundle0\") pod \"keda-operator-579c8958f7-7l7z4\" (UID: \"de195ee8-17c8-401d-a050-5b63216be373\") " pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:06.066361 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.066335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42g5d\" (UniqueName: \"kubernetes.io/projected/de195ee8-17c8-401d-a050-5b63216be373-kube-api-access-42g5d\") pod \"keda-operator-579c8958f7-7l7z4\" (UID: \"de195ee8-17c8-401d-a050-5b63216be373\") " pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:06.267116 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.267084    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"]
Sep 17 18:35:06.282729 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.282699    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"]
Sep 17 18:35:06.282872 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.282806    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:06.286520 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.286498    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 18:35:06.356072 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.355970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7c3f273b-550a-4f42-b16f-abc2632c3cc6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-c7v4j\" (UID: \"7c3f273b-550a-4f42-b16f-abc2632c3cc6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:06.356072 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.356021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fsm\" (UniqueName: \"kubernetes.io/projected/7c3f273b-550a-4f42-b16f-abc2632c3cc6-kube-api-access-q7fsm\") pod \"keda-metrics-apiserver-77857b7b98-c7v4j\" (UID: \"7c3f273b-550a-4f42-b16f-abc2632c3cc6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:06.356072 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.356044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7c3f273b-550a-4f42-b16f-abc2632c3cc6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-c7v4j\" (UID: \"7c3f273b-550a-4f42-b16f-abc2632c3cc6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:06.456868 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.456829    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7c3f273b-550a-4f42-b16f-abc2632c3cc6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-c7v4j\" (UID: \"7c3f273b-550a-4f42-b16f-abc2632c3cc6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:06.457069 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.456984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7c3f273b-550a-4f42-b16f-abc2632c3cc6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-c7v4j\" (UID: \"7c3f273b-550a-4f42-b16f-abc2632c3cc6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:06.457069 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.457024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fsm\" (UniqueName: \"kubernetes.io/projected/7c3f273b-550a-4f42-b16f-abc2632c3cc6-kube-api-access-q7fsm\") pod \"keda-metrics-apiserver-77857b7b98-c7v4j\" (UID: \"7c3f273b-550a-4f42-b16f-abc2632c3cc6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:06.457426 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.457398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/7c3f273b-550a-4f42-b16f-abc2632c3cc6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-c7v4j\" (UID: \"7c3f273b-550a-4f42-b16f-abc2632c3cc6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:06.459770 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.459741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7c3f273b-550a-4f42-b16f-abc2632c3cc6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-c7v4j\" (UID: \"7c3f273b-550a-4f42-b16f-abc2632c3cc6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:06.469253 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.469223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fsm\" (UniqueName: \"kubernetes.io/projected/7c3f273b-550a-4f42-b16f-abc2632c3cc6-kube-api-access-q7fsm\") pod \"keda-metrics-apiserver-77857b7b98-c7v4j\" (UID: \"7c3f273b-550a-4f42-b16f-abc2632c3cc6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:06.557916 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.557842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de195ee8-17c8-401d-a050-5b63216be373-certificates\") pod \"keda-operator-579c8958f7-7l7z4\" (UID: \"de195ee8-17c8-401d-a050-5b63216be373\") " pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:06.560750 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.560702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/de195ee8-17c8-401d-a050-5b63216be373-certificates\") pod \"keda-operator-579c8958f7-7l7z4\" (UID: \"de195ee8-17c8-401d-a050-5b63216be373\") " pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:06.594825 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.594780    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:06.757106 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.757070    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:06.912600 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:06.912513    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"]
Sep 17 18:35:06.917379 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:35:06.917351    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3f273b_550a_4f42_b16f_abc2632c3cc6.slice/crio-3fc21f7da3b7ebf4b7b378f3a2b0f65617b623ab1f8038a46f8d3d35b65489cc WatchSource:0}: Error finding container 3fc21f7da3b7ebf4b7b378f3a2b0f65617b623ab1f8038a46f8d3d35b65489cc: Status 404 returned error can't find the container with id 3fc21f7da3b7ebf4b7b378f3a2b0f65617b623ab1f8038a46f8d3d35b65489cc
Sep 17 18:35:07.087640 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:07.087604    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7l7z4"]
Sep 17 18:35:07.088246 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:35:07.088207    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde195ee8_17c8_401d_a050_5b63216be373.slice/crio-ced1a8f2f84ca0155b7bef18f816617c213ea603d31a2ab3edc65343e92adf0d WatchSource:0}: Error finding container ced1a8f2f84ca0155b7bef18f816617c213ea603d31a2ab3edc65343e92adf0d: Status 404 returned error can't find the container with id ced1a8f2f84ca0155b7bef18f816617c213ea603d31a2ab3edc65343e92adf0d
Sep 17 18:35:07.774252 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:07.774215    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-7l7z4" event={"ID":"de195ee8-17c8-401d-a050-5b63216be373","Type":"ContainerStarted","Data":"ced1a8f2f84ca0155b7bef18f816617c213ea603d31a2ab3edc65343e92adf0d"}
Sep 17 18:35:07.775633 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:07.775606    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j" event={"ID":"7c3f273b-550a-4f42-b16f-abc2632c3cc6","Type":"ContainerStarted","Data":"3fc21f7da3b7ebf4b7b378f3a2b0f65617b623ab1f8038a46f8d3d35b65489cc"}
Sep 17 18:35:11.797455 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:11.797418    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-7l7z4" event={"ID":"de195ee8-17c8-401d-a050-5b63216be373","Type":"ContainerStarted","Data":"9ad2ee2da47af63d8501e0a9f28b48b4fa86157926811393cd7195b61d7805ff"}
Sep 17 18:35:11.797878 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:11.797642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:11.799168 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:11.799136    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j" event={"ID":"7c3f273b-550a-4f42-b16f-abc2632c3cc6","Type":"ContainerStarted","Data":"afd6078f1d82fa5f500d26c69c8162c257d3e5a5c623995e38e88517b9946980"}
Sep 17 18:35:11.799293 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:11.799202    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:11.816763 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:11.816695    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-7l7z4" podStartSLOduration=2.683857463 podStartE2EDuration="6.816674167s" totalImagesPullingTime="4.132816704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:11.815384103 +0000 UTC m=+345.739166764" watchObservedRunningTime="2026-09-17 18:35:11.816674167 +0000 UTC m=+345.740456836"
Sep 17 18:35:11.839156 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:11.839097    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j" podStartSLOduration=1.535335993 podStartE2EDuration="5.839080671s" totalImagesPullingTime="4.303744678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:11.838059099 +0000 UTC m=+345.761841761" watchObservedRunningTime="2026-09-17 18:35:11.839080671 +0000 UTC m=+345.762863332"
Sep 17 18:35:22.808440 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:22.808409    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c7v4j"
Sep 17 18:35:26.769754 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:26.769719    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-fwhv7"
Sep 17 18:35:32.806277 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:32.806248    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-7l7z4"
Sep 17 18:35:37.680897 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.680847    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-97876576-j7zsz"]
Sep 17 18:35:37.687920 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.687892    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.705279 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.705248    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-97876576-j7zsz"]
Sep 17 18:35:37.755208 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.755170    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-trusted-ca-bundle\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.755411 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.755250    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-console-serving-cert\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.755411 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.755329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-console-oauth-config\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.755411 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.755365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwvwn\" (UniqueName: \"kubernetes.io/projected/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-kube-api-access-rwvwn\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.755411 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.755394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-service-ca\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.755623 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.755437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-console-config\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.755623 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.755460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-oauth-serving-cert\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.856595 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.856545    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-console-serving-cert\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.856789 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.856620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-console-oauth-config\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.856789 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.856649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwvwn\" (UniqueName: \"kubernetes.io/projected/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-kube-api-access-rwvwn\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.856789 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.856674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-service-ca\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.856789 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.856713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-console-config\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.856789 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.856734    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-oauth-serving-cert\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.856789 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.856766    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-trusted-ca-bundle\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.857468 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.857432    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-console-config\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.857468 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.857432    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-service-ca\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.857629 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.857582    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-trusted-ca-bundle\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.857775 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.857751    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-oauth-serving-cert\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.859497 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.859475    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-console-serving-cert\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.859580 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.859499    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-console-oauth-config\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.864389 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.864356    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwvwn\" (UniqueName: \"kubernetes.io/projected/8a5d3b4d-6145-406a-b3e7-06697ecbb3bc-kube-api-access-rwvwn\") pod \"console-97876576-j7zsz\" (UID: \"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc\") " pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:37.996532 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:37.996440    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:38.268777 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:38.268685    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-97876576-j7zsz"]
Sep 17 18:35:38.272197 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:35:38.272170    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5d3b4d_6145_406a_b3e7_06697ecbb3bc.slice/crio-eac7766c52d298d266ec11ad47ed6fa84928c5fb374a322baa3c686206dadabf WatchSource:0}: Error finding container eac7766c52d298d266ec11ad47ed6fa84928c5fb374a322baa3c686206dadabf: Status 404 returned error can't find the container with id eac7766c52d298d266ec11ad47ed6fa84928c5fb374a322baa3c686206dadabf
Sep 17 18:35:38.931900 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:38.931844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97876576-j7zsz" event={"ID":"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc","Type":"ContainerStarted","Data":"845f1127561eb06030c0660fb758300471b87e5be278426bd9da8d6dd5b98358"}
Sep 17 18:35:38.932306 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:38.931924    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-97876576-j7zsz" event={"ID":"8a5d3b4d-6145-406a-b3e7-06697ecbb3bc","Type":"ContainerStarted","Data":"eac7766c52d298d266ec11ad47ed6fa84928c5fb374a322baa3c686206dadabf"}
Sep 17 18:35:38.950391 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:38.950335    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-97876576-j7zsz" podStartSLOduration=1.950318931 podStartE2EDuration="1.950318931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:38.948345214 +0000 UTC m=+372.872127877" watchObservedRunningTime="2026-09-17 18:35:38.950318931 +0000 UTC m=+372.874101592"
Sep 17 18:35:47.996779 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:47.996741    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:47.997279 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:47.996842    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:48.002157 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:48.002132    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:48.978965 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:48.978934    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-97876576-j7zsz"
Sep 17 18:35:49.027952 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:35:49.027920    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc8cd6744-9w88f"]
Sep 17 18:36:14.048196 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.048123    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bc8cd6744-9w88f" podUID="6a5a219b-4f85-4188-bb52-7ba81ab89896" containerName="console" containerID="cri-o://4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b" gracePeriod=15
Sep 17 18:36:14.292513 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.292484    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:36:14.376142 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376065    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-trusted-ca-bundle\") pod \"6a5a219b-4f85-4188-bb52-7ba81ab89896\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") "
Sep 17 18:36:14.376142 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376103    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-serving-cert\") pod \"6a5a219b-4f85-4188-bb52-7ba81ab89896\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") "
Sep 17 18:36:14.376142 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376128    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-service-ca\") pod \"6a5a219b-4f85-4188-bb52-7ba81ab89896\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") "
Sep 17 18:36:14.376406 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376162    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-config\") pod \"6a5a219b-4f85-4188-bb52-7ba81ab89896\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") "
Sep 17 18:36:14.376406 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376288    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m8rp\" (UniqueName: \"kubernetes.io/projected/6a5a219b-4f85-4188-bb52-7ba81ab89896-kube-api-access-9m8rp\") pod \"6a5a219b-4f85-4188-bb52-7ba81ab89896\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") "
Sep 17 18:36:14.376406 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376349    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-oauth-serving-cert\") pod \"6a5a219b-4f85-4188-bb52-7ba81ab89896\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") "
Sep 17 18:36:14.376549 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376436    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-oauth-config\") pod \"6a5a219b-4f85-4188-bb52-7ba81ab89896\" (UID: \"6a5a219b-4f85-4188-bb52-7ba81ab89896\") "
Sep 17 18:36:14.376591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376556    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-config" (OuterVolumeSpecName: "console-config") pod "6a5a219b-4f85-4188-bb52-7ba81ab89896" (UID: "6a5a219b-4f85-4188-bb52-7ba81ab89896"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:14.376657 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376626    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6a5a219b-4f85-4188-bb52-7ba81ab89896" (UID: "6a5a219b-4f85-4188-bb52-7ba81ab89896"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:14.376846 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376821    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-service-ca" (OuterVolumeSpecName: "service-ca") pod "6a5a219b-4f85-4188-bb52-7ba81ab89896" (UID: "6a5a219b-4f85-4188-bb52-7ba81ab89896"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:14.376949 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376851    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-trusted-ca-bundle\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:36:14.376949 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376873    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:36:14.376949 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.376939    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6a5a219b-4f85-4188-bb52-7ba81ab89896" (UID: "6a5a219b-4f85-4188-bb52-7ba81ab89896"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:14.378734 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.378709    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5a219b-4f85-4188-bb52-7ba81ab89896-kube-api-access-9m8rp" (OuterVolumeSpecName: "kube-api-access-9m8rp") pod "6a5a219b-4f85-4188-bb52-7ba81ab89896" (UID: "6a5a219b-4f85-4188-bb52-7ba81ab89896"). InnerVolumeSpecName "kube-api-access-9m8rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:14.378867 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.378752    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6a5a219b-4f85-4188-bb52-7ba81ab89896" (UID: "6a5a219b-4f85-4188-bb52-7ba81ab89896"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:36:14.379190 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.379169    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6a5a219b-4f85-4188-bb52-7ba81ab89896" (UID: "6a5a219b-4f85-4188-bb52-7ba81ab89896"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:36:14.477360 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.477322    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-serving-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:36:14.477360 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.477354    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-service-ca\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:36:14.477360 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.477363    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m8rp\" (UniqueName: \"kubernetes.io/projected/6a5a219b-4f85-4188-bb52-7ba81ab89896-kube-api-access-9m8rp\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:36:14.477599 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.477372    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a5a219b-4f85-4188-bb52-7ba81ab89896-oauth-serving-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:36:14.477599 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:14.477381    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a5a219b-4f85-4188-bb52-7ba81ab89896-console-oauth-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:36:15.091119 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:15.091085    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a5a219b-4f85-4188-bb52-7ba81ab89896" containerID="4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b" exitCode=0
Sep 17 18:36:15.091522 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:15.091129    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc8cd6744-9w88f"
Sep 17 18:36:15.091522 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:15.091177    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc8cd6744-9w88f" event={"ID":"6a5a219b-4f85-4188-bb52-7ba81ab89896","Type":"ContainerDied","Data":"4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b"}
Sep 17 18:36:15.091522 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:15.091228    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc8cd6744-9w88f" event={"ID":"6a5a219b-4f85-4188-bb52-7ba81ab89896","Type":"ContainerDied","Data":"45a9cead9f6d5f2c96ed7a222e9b36b4e53cd637c51e2034948c2d3ae465ec96"}
Sep 17 18:36:15.091522 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:15.091250    2582 scope.go:117] "RemoveContainer" containerID="4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b"
Sep 17 18:36:15.100198 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:15.100180    2582 scope.go:117] "RemoveContainer" containerID="4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b"
Sep 17 18:36:15.100472 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:36:15.100457    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b\": container with ID starting with 4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b not found: ID does not exist" containerID="4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b"
Sep 17 18:36:15.100509 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:15.100482    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b"} err="failed to get container status \"4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b\": rpc error: code = NotFound desc = could not find container \"4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b\": container with ID starting with 4d00d32420cf1b5b2b0465699c0ca1f6b36617336a0384af3cd58c5426c2ed4b not found: ID does not exist"
Sep 17 18:36:15.112365 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:15.112339    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc8cd6744-9w88f"]
Sep 17 18:36:15.114791 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:15.114768    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bc8cd6744-9w88f"]
Sep 17 18:36:16.181951 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:16.181914    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a5a219b-4f85-4188-bb52-7ba81ab89896" path="/var/lib/kubelet/pods/6a5a219b-4f85-4188-bb52-7ba81ab89896/volumes"
Sep 17 18:36:29.918318 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.918281    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-mh7b5"]
Sep 17 18:36:29.918776 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.918605    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5a219b-4f85-4188-bb52-7ba81ab89896" containerName="console"
Sep 17 18:36:29.918776 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.918615    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5a219b-4f85-4188-bb52-7ba81ab89896" containerName="console"
Sep 17 18:36:29.918776 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.918671    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5a219b-4f85-4188-bb52-7ba81ab89896" containerName="console"
Sep 17 18:36:29.922867 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.922845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:36:29.928644 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.928615    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 18:36:29.928817 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.928684    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:36:29.928817 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.928633    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:36:29.929018 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.929000    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-dfcfz\""
Sep 17 18:36:29.953952 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.953918    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-mh7b5"]
Sep 17 18:36:29.970792 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.970760    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-t7jdl"]
Sep 17 18:36:29.974080 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.974057    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-t7jdl"
Sep 17 18:36:29.976404 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.976383    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:36:29.976404 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.976399    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-xblxv\""
Sep 17 18:36:29.987298 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:29.987272    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-t7jdl"]
Sep 17 18:36:30.106347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.106311    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlwwj\" (UniqueName: \"kubernetes.io/projected/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-kube-api-access-qlwwj\") pod \"kserve-controller-manager-69d59d7cdd-mh7b5\" (UID: \"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3\") " pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:36:30.106347 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.106352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4tk\" (UniqueName: \"kubernetes.io/projected/9b7741da-1844-4b58-b52b-109a9e69560a-kube-api-access-db4tk\") pod \"seaweedfs-64568bcd49-t7jdl\" (UID: \"9b7741da-1844-4b58-b52b-109a9e69560a\") " pod="kserve/seaweedfs-64568bcd49-t7jdl"
Sep 17 18:36:30.106586 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.106377    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9b7741da-1844-4b58-b52b-109a9e69560a-data\") pod \"seaweedfs-64568bcd49-t7jdl\" (UID: \"9b7741da-1844-4b58-b52b-109a9e69560a\") " pod="kserve/seaweedfs-64568bcd49-t7jdl"
Sep 17 18:36:30.106586 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.106488    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-cert\") pod \"kserve-controller-manager-69d59d7cdd-mh7b5\" (UID: \"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3\") " pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:36:30.207599 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.207509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-cert\") pod \"kserve-controller-manager-69d59d7cdd-mh7b5\" (UID: \"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3\") " pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:36:30.207599 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.207566    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlwwj\" (UniqueName: \"kubernetes.io/projected/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-kube-api-access-qlwwj\") pod \"kserve-controller-manager-69d59d7cdd-mh7b5\" (UID: \"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3\") " pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:36:30.207599 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.207594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4tk\" (UniqueName: \"kubernetes.io/projected/9b7741da-1844-4b58-b52b-109a9e69560a-kube-api-access-db4tk\") pod \"seaweedfs-64568bcd49-t7jdl\" (UID: \"9b7741da-1844-4b58-b52b-109a9e69560a\") " pod="kserve/seaweedfs-64568bcd49-t7jdl"
Sep 17 18:36:30.207912 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.207621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9b7741da-1844-4b58-b52b-109a9e69560a-data\") pod \"seaweedfs-64568bcd49-t7jdl\" (UID: \"9b7741da-1844-4b58-b52b-109a9e69560a\") " pod="kserve/seaweedfs-64568bcd49-t7jdl"
Sep 17 18:36:30.207986 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.207971    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9b7741da-1844-4b58-b52b-109a9e69560a-data\") pod \"seaweedfs-64568bcd49-t7jdl\" (UID: \"9b7741da-1844-4b58-b52b-109a9e69560a\") " pod="kserve/seaweedfs-64568bcd49-t7jdl"
Sep 17 18:36:30.210174 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.210153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-cert\") pod \"kserve-controller-manager-69d59d7cdd-mh7b5\" (UID: \"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3\") " pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:36:30.222764 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.222736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlwwj\" (UniqueName: \"kubernetes.io/projected/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-kube-api-access-qlwwj\") pod \"kserve-controller-manager-69d59d7cdd-mh7b5\" (UID: \"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3\") " pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:36:30.224620 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.224599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4tk\" (UniqueName: \"kubernetes.io/projected/9b7741da-1844-4b58-b52b-109a9e69560a-kube-api-access-db4tk\") pod \"seaweedfs-64568bcd49-t7jdl\" (UID: \"9b7741da-1844-4b58-b52b-109a9e69560a\") " pod="kserve/seaweedfs-64568bcd49-t7jdl"
Sep 17 18:36:30.233324 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.233300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:36:30.284470 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.284410    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-t7jdl"
Sep 17 18:36:30.560672 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.560639    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-mh7b5"]
Sep 17 18:36:30.564380 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:36:30.564347    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4239c84_e4e3_49df_9ee8_8f23ab67c5c3.slice/crio-6299514f47b47f9fb0a405b5539aa30f388ae2a486fe8d4838e1d478d367f8a1 WatchSource:0}: Error finding container 6299514f47b47f9fb0a405b5539aa30f388ae2a486fe8d4838e1d478d367f8a1: Status 404 returned error can't find the container with id 6299514f47b47f9fb0a405b5539aa30f388ae2a486fe8d4838e1d478d367f8a1
Sep 17 18:36:30.612059 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:36:30.612029    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7741da_1844_4b58_b52b_109a9e69560a.slice/crio-ed7eacec2747c1eda5da35b4fd5a78d9ba76efee604e52f5cff24315c58b1c29 WatchSource:0}: Error finding container ed7eacec2747c1eda5da35b4fd5a78d9ba76efee604e52f5cff24315c58b1c29: Status 404 returned error can't find the container with id ed7eacec2747c1eda5da35b4fd5a78d9ba76efee604e52f5cff24315c58b1c29
Sep 17 18:36:30.618077 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:30.618052    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-t7jdl"]
Sep 17 18:36:31.162328 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:31.162287    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-t7jdl" event={"ID":"9b7741da-1844-4b58-b52b-109a9e69560a","Type":"ContainerStarted","Data":"ed7eacec2747c1eda5da35b4fd5a78d9ba76efee604e52f5cff24315c58b1c29"}
Sep 17 18:36:31.163783 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:31.163745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5" event={"ID":"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3","Type":"ContainerStarted","Data":"6299514f47b47f9fb0a405b5539aa30f388ae2a486fe8d4838e1d478d367f8a1"}
Sep 17 18:36:35.186342 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:35.186304    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-t7jdl" event={"ID":"9b7741da-1844-4b58-b52b-109a9e69560a","Type":"ContainerStarted","Data":"db66aebd962c35d79655924226dc6dcd47409873cad59d616c46be6cd8647362"}
Sep 17 18:36:35.186767 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:35.186426    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-t7jdl"
Sep 17 18:36:35.187915 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:35.187892    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5" event={"ID":"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3","Type":"ContainerStarted","Data":"3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776"}
Sep 17 18:36:35.187993 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:35.187937    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:36:35.215944 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:35.215858    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-t7jdl" podStartSLOduration=2.445961467 podStartE2EDuration="6.215840885s" totalImagesPullingTime="3.769879418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:35.214816509 +0000 UTC m=+429.138599172" watchObservedRunningTime="2026-09-17 18:36:35.215840885 +0000 UTC m=+429.139623595"
Sep 17 18:36:35.239291 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:35.239197    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5" podStartSLOduration=2.513986139 podStartE2EDuration="6.239182421s" totalImagesPullingTime="3.725196282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:35.238814207 +0000 UTC m=+429.162596867" watchObservedRunningTime="2026-09-17 18:36:35.239182421 +0000 UTC m=+429.162965081"
Sep 17 18:36:41.194467 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:36:41.194382    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-t7jdl"
Sep 17 18:37:06.197132 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:06.197097    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:37:14.256499 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.256461    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-mh7b5"]
Sep 17 18:37:14.257058 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.256685    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5" podUID="f4239c84-e4e3-49df-9ee8-8f23ab67c5c3" containerName="manager" containerID="cri-o://3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776" gracePeriod=10
Sep 17 18:37:14.277124 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.277095    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-cbhgq"]
Sep 17 18:37:14.279299 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.279284    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq"
Sep 17 18:37:14.289596 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.289570    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-cbhgq"]
Sep 17 18:37:14.405812 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.405776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75fdea0a-c437-4815-8060-0251f13a2bc8-cert\") pod \"kserve-controller-manager-69d59d7cdd-cbhgq\" (UID: \"75fdea0a-c437-4815-8060-0251f13a2bc8\") " pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq"
Sep 17 18:37:14.405812 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.405816    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ppw\" (UniqueName: \"kubernetes.io/projected/75fdea0a-c437-4815-8060-0251f13a2bc8-kube-api-access-b6ppw\") pod \"kserve-controller-manager-69d59d7cdd-cbhgq\" (UID: \"75fdea0a-c437-4815-8060-0251f13a2bc8\") " pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq"
Sep 17 18:37:14.497613 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.497582    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:37:14.507227 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.507159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75fdea0a-c437-4815-8060-0251f13a2bc8-cert\") pod \"kserve-controller-manager-69d59d7cdd-cbhgq\" (UID: \"75fdea0a-c437-4815-8060-0251f13a2bc8\") " pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq"
Sep 17 18:37:14.507227 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.507196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ppw\" (UniqueName: \"kubernetes.io/projected/75fdea0a-c437-4815-8060-0251f13a2bc8-kube-api-access-b6ppw\") pod \"kserve-controller-manager-69d59d7cdd-cbhgq\" (UID: \"75fdea0a-c437-4815-8060-0251f13a2bc8\") " pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq"
Sep 17 18:37:14.509631 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.509608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75fdea0a-c437-4815-8060-0251f13a2bc8-cert\") pod \"kserve-controller-manager-69d59d7cdd-cbhgq\" (UID: \"75fdea0a-c437-4815-8060-0251f13a2bc8\") " pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq"
Sep 17 18:37:14.518374 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.518335    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ppw\" (UniqueName: \"kubernetes.io/projected/75fdea0a-c437-4815-8060-0251f13a2bc8-kube-api-access-b6ppw\") pod \"kserve-controller-manager-69d59d7cdd-cbhgq\" (UID: \"75fdea0a-c437-4815-8060-0251f13a2bc8\") " pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq"
Sep 17 18:37:14.608141 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.608108    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-cert\") pod \"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3\" (UID: \"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3\") "
Sep 17 18:37:14.608303 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.608206    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlwwj\" (UniqueName: \"kubernetes.io/projected/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-kube-api-access-qlwwj\") pod \"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3\" (UID: \"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3\") "
Sep 17 18:37:14.610408 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.610375    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-cert" (OuterVolumeSpecName: "cert") pod "f4239c84-e4e3-49df-9ee8-8f23ab67c5c3" (UID: "f4239c84-e4e3-49df-9ee8-8f23ab67c5c3"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:37:14.610523 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.610410    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-kube-api-access-qlwwj" (OuterVolumeSpecName: "kube-api-access-qlwwj") pod "f4239c84-e4e3-49df-9ee8-8f23ab67c5c3" (UID: "f4239c84-e4e3-49df-9ee8-8f23ab67c5c3"). InnerVolumeSpecName "kube-api-access-qlwwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:14.633619 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.633591    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq"
Sep 17 18:37:14.709199 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.709166    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlwwj\" (UniqueName: \"kubernetes.io/projected/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-kube-api-access-qlwwj\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:37:14.709199 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.709198    2582 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:37:14.901452 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:14.901424    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-cbhgq"]
Sep 17 18:37:14.903676 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:37:14.903631    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75fdea0a_c437_4815_8060_0251f13a2bc8.slice/crio-7222970400bed8e8c7064dc431b25d54b3ce6d96ceb8e7869ffeb5cb83d1b23c WatchSource:0}: Error finding container 7222970400bed8e8c7064dc431b25d54b3ce6d96ceb8e7869ffeb5cb83d1b23c: Status 404 returned error can't find the container with id 7222970400bed8e8c7064dc431b25d54b3ce6d96ceb8e7869ffeb5cb83d1b23c
Sep 17 18:37:15.370961 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:15.370921    2582 generic.go:356] "Generic (PLEG): container finished" podID="f4239c84-e4e3-49df-9ee8-8f23ab67c5c3" containerID="3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776" exitCode=0
Sep 17 18:37:15.371306 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:15.371018    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5"
Sep 17 18:37:15.371306 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:15.371020    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5" event={"ID":"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3","Type":"ContainerDied","Data":"3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776"}
Sep 17 18:37:15.371306 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:15.371153    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-mh7b5" event={"ID":"f4239c84-e4e3-49df-9ee8-8f23ab67c5c3","Type":"ContainerDied","Data":"6299514f47b47f9fb0a405b5539aa30f388ae2a486fe8d4838e1d478d367f8a1"}
Sep 17 18:37:15.371306 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:15.371187    2582 scope.go:117] "RemoveContainer" containerID="3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776"
Sep 17 18:37:15.372336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:15.372218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq" event={"ID":"75fdea0a-c437-4815-8060-0251f13a2bc8","Type":"ContainerStarted","Data":"7222970400bed8e8c7064dc431b25d54b3ce6d96ceb8e7869ffeb5cb83d1b23c"}
Sep 17 18:37:15.380002 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:15.379976    2582 scope.go:117] "RemoveContainer" containerID="3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776"
Sep 17 18:37:15.380292 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:37:15.380271    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776\": container with ID starting with 3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776 not found: ID does not exist" containerID="3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776"
Sep 17 18:37:15.380350 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:15.380308    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776"} err="failed to get container status \"3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776\": rpc error: code = NotFound desc = could not find container \"3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776\": container with ID starting with 3371f9030551561f917bb568e6234663ed54fb6dc839235c7e9c299ad22a3776 not found: ID does not exist"
Sep 17 18:37:15.390632 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:15.390608    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-mh7b5"]
Sep 17 18:37:15.394446 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:15.394411    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-69d59d7cdd-mh7b5"]
Sep 17 18:37:16.182137 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:16.182104    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4239c84-e4e3-49df-9ee8-8f23ab67c5c3" path="/var/lib/kubelet/pods/f4239c84-e4e3-49df-9ee8-8f23ab67c5c3/volumes"
Sep 17 18:37:16.379382 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:16.379345    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq" event={"ID":"75fdea0a-c437-4815-8060-0251f13a2bc8","Type":"ContainerStarted","Data":"5974907348dda11675cf1f4ed2afb8ddb11818da5464d6f2f0f93d6d4b0fde9f"}
Sep 17 18:37:16.379800 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:16.379430    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq"
Sep 17 18:37:16.394793 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:16.394734    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq" podStartSLOduration=1.9809671899999999 podStartE2EDuration="2.394716309s" totalImagesPullingTime="413.749119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:16.393629327 +0000 UTC m=+470.317411989" watchObservedRunningTime="2026-09-17 18:37:16.394716309 +0000 UTC m=+470.318498969"
Sep 17 18:37:47.390660 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:47.390622    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-69d59d7cdd-cbhgq"
Sep 17 18:37:48.706335 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.706291    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-2vfps"]
Sep 17 18:37:48.706841 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.706825    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4239c84-e4e3-49df-9ee8-8f23ab67c5c3" containerName="manager"
Sep 17 18:37:48.706903 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.706845    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4239c84-e4e3-49df-9ee8-8f23ab67c5c3" containerName="manager"
Sep 17 18:37:48.706968 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.706955    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4239c84-e4e3-49df-9ee8-8f23ab67c5c3" containerName="manager"
Sep 17 18:37:48.710362 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.710338    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:48.712997 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.712969    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 18:37:48.713159 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.713138    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 18:37:48.713226 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.713172    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-n745q\""
Sep 17 18:37:48.719302 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.719273    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-2vfps"]
Sep 17 18:37:48.810298 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.810255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-metrics-certs\") pod \"odh-model-controller-7b966d7984-2vfps\" (UID: \"0cca4acc-60f9-4160-a8e3-46b862f27304\") " pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:48.810298 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.810302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9s97\" (UniqueName: \"kubernetes.io/projected/0cca4acc-60f9-4160-a8e3-46b862f27304-kube-api-access-c9s97\") pod \"odh-model-controller-7b966d7984-2vfps\" (UID: \"0cca4acc-60f9-4160-a8e3-46b862f27304\") " pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:48.810566 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.810377    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-cert\") pod \"odh-model-controller-7b966d7984-2vfps\" (UID: \"0cca4acc-60f9-4160-a8e3-46b862f27304\") " pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:48.910956 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.910919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-metrics-certs\") pod \"odh-model-controller-7b966d7984-2vfps\" (UID: \"0cca4acc-60f9-4160-a8e3-46b862f27304\") " pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:48.910956 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.910957    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9s97\" (UniqueName: \"kubernetes.io/projected/0cca4acc-60f9-4160-a8e3-46b862f27304-kube-api-access-c9s97\") pod \"odh-model-controller-7b966d7984-2vfps\" (UID: \"0cca4acc-60f9-4160-a8e3-46b862f27304\") " pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:48.911172 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.911037    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-cert\") pod \"odh-model-controller-7b966d7984-2vfps\" (UID: \"0cca4acc-60f9-4160-a8e3-46b862f27304\") " pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:48.911172 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:37:48.911090    2582 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 18:37:48.911172 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:37:48.911153    2582 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 18:37:48.911279 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:37:48.911174    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-metrics-certs podName:0cca4acc-60f9-4160-a8e3-46b862f27304 nodeName:}" failed. No retries permitted until 2026-09-17 18:37:49.411151248 +0000 UTC m=+503.334933892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-metrics-certs") pod "odh-model-controller-7b966d7984-2vfps" (UID: "0cca4acc-60f9-4160-a8e3-46b862f27304") : secret "odh-model-controller-metrics-tls" not found
Sep 17 18:37:48.911279 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:37:48.911212    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-cert podName:0cca4acc-60f9-4160-a8e3-46b862f27304 nodeName:}" failed. No retries permitted until 2026-09-17 18:37:49.411195697 +0000 UTC m=+503.334978340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-cert") pod "odh-model-controller-7b966d7984-2vfps" (UID: "0cca4acc-60f9-4160-a8e3-46b862f27304") : secret "odh-model-controller-webhook-cert" not found
Sep 17 18:37:48.919564 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:48.919532    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9s97\" (UniqueName: \"kubernetes.io/projected/0cca4acc-60f9-4160-a8e3-46b862f27304-kube-api-access-c9s97\") pod \"odh-model-controller-7b966d7984-2vfps\" (UID: \"0cca4acc-60f9-4160-a8e3-46b862f27304\") " pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:49.414326 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:49.414289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-metrics-certs\") pod \"odh-model-controller-7b966d7984-2vfps\" (UID: \"0cca4acc-60f9-4160-a8e3-46b862f27304\") " pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:49.414517 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:49.414383    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-cert\") pod \"odh-model-controller-7b966d7984-2vfps\" (UID: \"0cca4acc-60f9-4160-a8e3-46b862f27304\") " pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:49.417027 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:49.416992    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-metrics-certs\") pod \"odh-model-controller-7b966d7984-2vfps\" (UID: \"0cca4acc-60f9-4160-a8e3-46b862f27304\") " pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:49.417152 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:49.417113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cca4acc-60f9-4160-a8e3-46b862f27304-cert\") pod \"odh-model-controller-7b966d7984-2vfps\" (UID: \"0cca4acc-60f9-4160-a8e3-46b862f27304\") " pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:49.623422 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:49.623373    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:49.899334 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:49.899191    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-2vfps"]
Sep 17 18:37:49.901524 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:37:49.901495    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cca4acc_60f9_4160_a8e3_46b862f27304.slice/crio-7992a677a107c5aad9489c2902abf8afdbd380a457bbd12412421ab2a2853a4f WatchSource:0}: Error finding container 7992a677a107c5aad9489c2902abf8afdbd380a457bbd12412421ab2a2853a4f: Status 404 returned error can't find the container with id 7992a677a107c5aad9489c2902abf8afdbd380a457bbd12412421ab2a2853a4f
Sep 17 18:37:50.537615 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:50.537572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-2vfps" event={"ID":"0cca4acc-60f9-4160-a8e3-46b862f27304","Type":"ContainerStarted","Data":"7992a677a107c5aad9489c2902abf8afdbd380a457bbd12412421ab2a2853a4f"}
Sep 17 18:37:53.554369 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:53.554334    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-2vfps" event={"ID":"0cca4acc-60f9-4160-a8e3-46b862f27304","Type":"ContainerStarted","Data":"8c4a266fee667f00c1f73ac5c923403d860256e6f0313f3896ba4c34ceb0f302"}
Sep 17 18:37:53.554766 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:53.554471    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:37:53.571330 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:37:53.571279    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-2vfps" podStartSLOduration=2.935831051 podStartE2EDuration="5.57126294s" totalImagesPullingTime="2.635431889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:53.569280207 +0000 UTC m=+507.493062866" watchObservedRunningTime="2026-09-17 18:37:53.57126294 +0000 UTC m=+507.495045629"
Sep 17 18:38:04.561256 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:38:04.561222    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-2vfps"
Sep 17 18:39:35.683563 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.683524    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"]
Sep 17 18:39:35.687311 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.687294    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.691100 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.691076    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:39:35.691302 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.691284    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-f604d-kube-rbac-proxy-sar-config\""
Sep 17 18:39:35.692006 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.691988    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:39:35.692076 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.692004    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:39:35.692076 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.692017    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\""
Sep 17 18:39:35.692076 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.691989    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:39:35.692202 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.692003    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-f604d-predictor-serving-cert\""
Sep 17 18:39:35.705492 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.705461    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"]
Sep 17 18:39:35.873763 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.873725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.874028 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.873792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2jt\" (UniqueName: \"kubernetes.io/projected/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-api-access-6c2jt\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.874028 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.873854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.874028 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.873878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.874028 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.873956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.975369 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.975273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.975369 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.975343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2jt\" (UniqueName: \"kubernetes.io/projected/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-api-access-6c2jt\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.975588 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.975373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.975588 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.975401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.975588 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.975528    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.976156 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.976122    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.976309 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.976285    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.976373 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.976293    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.978330 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.978311    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.985437 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.985410    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2jt\" (UniqueName: \"kubernetes.io/projected/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-api-access-6c2jt\") pod \"isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:35.998051 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:35.998023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:39:36.282410 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:36.282330    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"]
Sep 17 18:39:36.287007 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:39:36.286977    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430119a4_c1d1_4ac1_b60e_f67a7ba8c77c.slice/crio-4a4f23c6e27d2bc3e217fc1dfa2b8c374735596fd45bbffe1ea2b8f8f2570fe4 WatchSource:0}: Error finding container 4a4f23c6e27d2bc3e217fc1dfa2b8c374735596fd45bbffe1ea2b8f8f2570fe4: Status 404 returned error can't find the container with id 4a4f23c6e27d2bc3e217fc1dfa2b8c374735596fd45bbffe1ea2b8f8f2570fe4
Sep 17 18:39:37.030491 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:37.030446    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" event={"ID":"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c","Type":"ContainerStarted","Data":"4a4f23c6e27d2bc3e217fc1dfa2b8c374735596fd45bbffe1ea2b8f8f2570fe4"}
Sep 17 18:39:40.048981 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:40.048946    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" event={"ID":"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c","Type":"ContainerStarted","Data":"03bdbb70b032675adfa07d21b37e58fb529ad4f12c4753d67b5e5ae33c0dcd4b"}
Sep 17 18:39:44.070050 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:44.070016    2582 generic.go:356] "Generic (PLEG): container finished" podID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerID="03bdbb70b032675adfa07d21b37e58fb529ad4f12c4753d67b5e5ae33c0dcd4b" exitCode=0
Sep 17 18:39:44.070439 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:44.070097    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" event={"ID":"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c","Type":"ContainerDied","Data":"03bdbb70b032675adfa07d21b37e58fb529ad4f12c4753d67b5e5ae33c0dcd4b"}
Sep 17 18:39:44.071270 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:44.071255    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:39:58.163036 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:39:58.162994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" event={"ID":"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c","Type":"ContainerStarted","Data":"fd69845393e7dc9ba022e6d8f7951915fe29ae75fd765c82413c75cea3b3e200"}
Sep 17 18:40:00.176070 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:00.175984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" event={"ID":"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c","Type":"ContainerStarted","Data":"dfe4e60cc9316ac32c10be7fe58994f1379e89734c23dd103b21ea6632dccbb7"}
Sep 17 18:40:03.193189 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:03.193150    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" event={"ID":"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c","Type":"ContainerStarted","Data":"5ca5d39ef29fecfdc598adfa557ed0050dbbdab0a3ade0048fd7e243d62b6d04"}
Sep 17 18:40:03.193670 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:03.193351    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:40:03.193670 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:03.193496    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:40:03.194807 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:03.194772    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:40:03.236793 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:03.236731    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podStartSLOduration=2.991963932 podStartE2EDuration="28.236712904s" totalImagesPullingTime="21.454285112s" totalInitContainerRuntime="3.79046386s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:39:35 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:03.233667691 +0000 UTC m=+637.157450352" watchObservedRunningTime="2026-09-17 18:40:03.236712904 +0000 UTC m=+637.160495565"
Sep 17 18:40:04.198532 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:04.198500    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:40:04.198972 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:04.198611    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:40:04.199766 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:04.199738    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:40:04.202637 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:04.202619    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:40:05.203745 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:05.203699    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:40:05.204209 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:05.204075    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:40:06.208217 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:06.208175    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:40:06.208631 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:06.208480    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:40:16.208608 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:16.208544    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:40:16.209134 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:16.208967    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:40:26.208400 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:26.208347    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:40:26.208930 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:26.208870    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:40:36.208992 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:36.208940    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:40:36.209505 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:36.209349    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:40:46.209007 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:46.208957    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:40:46.209561 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:46.209536    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:40:56.208742 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:56.208688    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:40:56.209308 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:40:56.209208    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:41:06.208599 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:06.208555    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:41:06.209230 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:06.209200    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:41:16.182530 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:16.182498    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:41:16.183073 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:16.182595    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:41:25.694763 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.694721    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"]
Sep 17 18:41:25.698482 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.698462    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.701246 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.701222    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:41:25.701385 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.701293    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:41:25.712863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.712836    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"]
Sep 17 18:41:25.758607 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.758576    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.758774 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.758628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.758774 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.758654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.758774 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.758685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.758774 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.758715    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdn59\" (UniqueName: \"kubernetes.io/projected/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-api-access-gdn59\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.859714 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.859675    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.859921 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.859743    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.859921 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.859770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.859921 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.859799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.859921 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.859835    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdn59\" (UniqueName: \"kubernetes.io/projected/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-api-access-gdn59\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.860295 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.860274    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.860666 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.860644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.860736 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.860644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.862702 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.862682    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-proxy-tls\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:25.869289 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:25.869261    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdn59\" (UniqueName: \"kubernetes.io/projected/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-api-access-gdn59\") pod \"isvc-canary-rollback-predictor-7679bbb844-thdfg\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:26.009472 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:26.009383    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:26.288441 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:26.288396    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"]
Sep 17 18:41:26.290797 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:41:26.290766    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7dfe44_51d5_4305_b7c2_a5414a185b3b.slice/crio-87a4b3883f7b89a6347acea82515e1c5dcaae6cac5d786e334b986fe62a909f0 WatchSource:0}: Error finding container 87a4b3883f7b89a6347acea82515e1c5dcaae6cac5d786e334b986fe62a909f0: Status 404 returned error can't find the container with id 87a4b3883f7b89a6347acea82515e1c5dcaae6cac5d786e334b986fe62a909f0
Sep 17 18:41:26.598736 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:26.598693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" event={"ID":"3f7dfe44-51d5-4305-b7c2-a5414a185b3b","Type":"ContainerStarted","Data":"a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5"}
Sep 17 18:41:26.598997 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:26.598781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" event={"ID":"3f7dfe44-51d5-4305-b7c2-a5414a185b3b","Type":"ContainerStarted","Data":"87a4b3883f7b89a6347acea82515e1c5dcaae6cac5d786e334b986fe62a909f0"}
Sep 17 18:41:30.622093 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:30.621996    2582 generic.go:356] "Generic (PLEG): container finished" podID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerID="a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5" exitCode=0
Sep 17 18:41:30.622093 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:30.622065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" event={"ID":"3f7dfe44-51d5-4305-b7c2-a5414a185b3b","Type":"ContainerDied","Data":"a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5"}
Sep 17 18:41:30.646714 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:30.646683    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"]
Sep 17 18:41:30.646985 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:30.646961    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" containerID="cri-o://fd69845393e7dc9ba022e6d8f7951915fe29ae75fd765c82413c75cea3b3e200" gracePeriod=30
Sep 17 18:41:30.647061 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:30.647022    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" containerID="cri-o://5ca5d39ef29fecfdc598adfa557ed0050dbbdab0a3ade0048fd7e243d62b6d04" gracePeriod=30
Sep 17 18:41:30.647118 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:30.647074    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kube-rbac-proxy" containerID="cri-o://dfe4e60cc9316ac32c10be7fe58994f1379e89734c23dd103b21ea6632dccbb7" gracePeriod=30
Sep 17 18:41:31.629164 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:31.629129    2582 generic.go:356] "Generic (PLEG): container finished" podID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerID="dfe4e60cc9316ac32c10be7fe58994f1379e89734c23dd103b21ea6632dccbb7" exitCode=2
Sep 17 18:41:31.629597 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:31.629212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" event={"ID":"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c","Type":"ContainerDied","Data":"dfe4e60cc9316ac32c10be7fe58994f1379e89734c23dd103b21ea6632dccbb7"}
Sep 17 18:41:31.631370 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:31.631345    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" event={"ID":"3f7dfe44-51d5-4305-b7c2-a5414a185b3b","Type":"ContainerStarted","Data":"270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e"}
Sep 17 18:41:31.631516 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:31.631404    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" event={"ID":"3f7dfe44-51d5-4305-b7c2-a5414a185b3b","Type":"ContainerStarted","Data":"58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059"}
Sep 17 18:41:31.631634 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:31.631615    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:31.654203 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:31.654150    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podStartSLOduration=2.674112874 podStartE2EDuration="6.654132008s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.980019134s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:31.652011849 +0000 UTC m=+725.575794518" watchObservedRunningTime="2026-09-17 18:41:31.654132008 +0000 UTC m=+725.577914705"
Sep 17 18:41:32.636372 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:32.636336    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:32.638066 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:32.638031    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:41:33.641275 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:33.641232    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:41:34.199101 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:34.199060    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:41:36.177539 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:36.177443    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:41:36.177946 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:36.177809    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:41:36.657774 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:36.657737    2582 generic.go:356] "Generic (PLEG): container finished" podID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerID="fd69845393e7dc9ba022e6d8f7951915fe29ae75fd765c82413c75cea3b3e200" exitCode=0
Sep 17 18:41:36.657972 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:36.657785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" event={"ID":"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c","Type":"ContainerDied","Data":"fd69845393e7dc9ba022e6d8f7951915fe29ae75fd765c82413c75cea3b3e200"}
Sep 17 18:41:38.646070 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:38.646040    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:41:38.646653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:38.646624    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:41:39.199498 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:39.199452    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:41:44.199119 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:44.199070    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:41:44.199537 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:44.199211    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:41:46.177285 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:46.177230    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:41:46.177767 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:46.177602    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:41:48.646960 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:48.646917    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:41:49.199709 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:49.199669    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:41:54.199325 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:54.199282    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:41:56.177358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:56.177313    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:41:56.177783 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:56.177722    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:41:56.183006 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:56.182980    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:41:56.183152 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:56.183084    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:41:58.647454 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:58.647411    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:41:59.199837 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:41:59.199790    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:42:00.776206 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.776175    2582 generic.go:356] "Generic (PLEG): container finished" podID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerID="5ca5d39ef29fecfdc598adfa557ed0050dbbdab0a3ade0048fd7e243d62b6d04" exitCode=0
Sep 17 18:42:00.776537 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.776245    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" event={"ID":"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c","Type":"ContainerDied","Data":"5ca5d39ef29fecfdc598adfa557ed0050dbbdab0a3ade0048fd7e243d62b6d04"}
Sep 17 18:42:00.794607 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.794581    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:42:00.865131 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.865101    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-rbac-proxy-sar-config\") pod \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") "
Sep 17 18:42:00.865336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.865186    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c2jt\" (UniqueName: \"kubernetes.io/projected/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-api-access-6c2jt\") pod \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") "
Sep 17 18:42:00.865336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.865215    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-cabundle-cert\") pod \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") "
Sep 17 18:42:00.865336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.865238    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-proxy-tls\") pod \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") "
Sep 17 18:42:00.865336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.865271    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kserve-provision-location\") pod \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\" (UID: \"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c\") "
Sep 17 18:42:00.865593 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.865561    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" (UID: "430119a4-c1d1-4ac1-b60e-f67a7ba8c77c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:00.865707 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.865687    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" (UID: "430119a4-c1d1-4ac1-b60e-f67a7ba8c77c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:00.865756 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.865730    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" (UID: "430119a4-c1d1-4ac1-b60e-f67a7ba8c77c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:00.867628 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.867604    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-api-access-6c2jt" (OuterVolumeSpecName: "kube-api-access-6c2jt") pod "430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" (UID: "430119a4-c1d1-4ac1-b60e-f67a7ba8c77c"). InnerVolumeSpecName "kube-api-access-6c2jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:00.867721 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.867621    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" (UID: "430119a4-c1d1-4ac1-b60e-f67a7ba8c77c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:00.966482 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.966395    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c2jt\" (UniqueName: \"kubernetes.io/projected/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-api-access-6c2jt\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:42:00.966482 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.966426    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-cabundle-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:42:00.966482 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.966435    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-proxy-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:42:00.966482 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.966444    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kserve-provision-location\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:42:00.966482 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:00.966453    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:42:01.783293 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:01.783256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx" event={"ID":"430119a4-c1d1-4ac1-b60e-f67a7ba8c77c","Type":"ContainerDied","Data":"4a4f23c6e27d2bc3e217fc1dfa2b8c374735596fd45bbffe1ea2b8f8f2570fe4"}
Sep 17 18:42:01.783293 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:01.783297    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"
Sep 17 18:42:01.783759 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:01.783329    2582 scope.go:117] "RemoveContainer" containerID="5ca5d39ef29fecfdc598adfa557ed0050dbbdab0a3ade0048fd7e243d62b6d04"
Sep 17 18:42:01.792626 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:01.792604    2582 scope.go:117] "RemoveContainer" containerID="dfe4e60cc9316ac32c10be7fe58994f1379e89734c23dd103b21ea6632dccbb7"
Sep 17 18:42:01.800294 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:01.800269    2582 scope.go:117] "RemoveContainer" containerID="fd69845393e7dc9ba022e6d8f7951915fe29ae75fd765c82413c75cea3b3e200"
Sep 17 18:42:01.806422 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:01.806381    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"]
Sep 17 18:42:01.809068 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:01.809015    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-f604d-predictor-7c79d88b85-kzptx"]
Sep 17 18:42:01.809146 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:01.809091    2582 scope.go:117] "RemoveContainer" containerID="03bdbb70b032675adfa07d21b37e58fb529ad4f12c4753d67b5e5ae33c0dcd4b"
Sep 17 18:42:02.182088 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:02.182053    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" path="/var/lib/kubelet/pods/430119a4-c1d1-4ac1-b60e-f67a7ba8c77c/volumes"
Sep 17 18:42:08.646795 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:08.646753    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:42:18.646874 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:18.646828    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:42:28.647119 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:28.647079    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:42:38.647539 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:38.647446    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:42:48.647675 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:48.647639    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:42:55.727971 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:55.727938    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"]
Sep 17 18:42:55.728413 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:55.728193    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container" containerID="cri-o://58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059" gracePeriod=30
Sep 17 18:42:55.728413 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:55.728248    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kube-rbac-proxy" containerID="cri-o://270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e" gracePeriod=30
Sep 17 18:42:56.040054 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:56.039968    2582 generic.go:356] "Generic (PLEG): container finished" podID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerID="270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e" exitCode=2
Sep 17 18:42:56.040208 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:56.040041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" event={"ID":"3f7dfe44-51d5-4305-b7c2-a5414a185b3b","Type":"ContainerDied","Data":"270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e"}
Sep 17 18:42:58.642213 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:58.642168    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 18:42:58.646534 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:42:58.646502    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:43:00.681582 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.681557    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:43:00.789091 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.789012    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-proxy-tls\") pod \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") "
Sep 17 18:43:00.789091 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.789058    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-cabundle-cert\") pod \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") "
Sep 17 18:43:00.789289 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.789095    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kserve-provision-location\") pod \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") "
Sep 17 18:43:00.789289 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.789112    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdn59\" (UniqueName: \"kubernetes.io/projected/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-api-access-gdn59\") pod \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") "
Sep 17 18:43:00.789289 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.789142    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-rbac-proxy-sar-config\") pod \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\" (UID: \"3f7dfe44-51d5-4305-b7c2-a5414a185b3b\") "
Sep 17 18:43:00.789464 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.789428    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f7dfe44-51d5-4305-b7c2-a5414a185b3b" (UID: "3f7dfe44-51d5-4305-b7c2-a5414a185b3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:00.789587 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.789542    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f7dfe44-51d5-4305-b7c2-a5414a185b3b" (UID: "3f7dfe44-51d5-4305-b7c2-a5414a185b3b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:00.789587 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.789564    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3f7dfe44-51d5-4305-b7c2-a5414a185b3b" (UID: "3f7dfe44-51d5-4305-b7c2-a5414a185b3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:00.791586 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.791561    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f7dfe44-51d5-4305-b7c2-a5414a185b3b" (UID: "3f7dfe44-51d5-4305-b7c2-a5414a185b3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:00.791586 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.791572    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-api-access-gdn59" (OuterVolumeSpecName: "kube-api-access-gdn59") pod "3f7dfe44-51d5-4305-b7c2-a5414a185b3b" (UID: "3f7dfe44-51d5-4305-b7c2-a5414a185b3b"). InnerVolumeSpecName "kube-api-access-gdn59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:00.890197 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.890160    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.890197 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.890187    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-proxy-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.890197 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.890197    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-cabundle-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.890197 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.890207    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kserve-provision-location\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.890499 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.890216    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdn59\" (UniqueName: \"kubernetes.io/projected/3f7dfe44-51d5-4305-b7c2-a5414a185b3b-kube-api-access-gdn59\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:43:00.988172 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988135    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"]
Sep 17 18:43:00.988551 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988533    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container"
Sep 17 18:43:00.988551 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988550    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988569    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988574    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988579    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="storage-initializer"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988585    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="storage-initializer"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988593    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988598    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988604    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kube-rbac-proxy"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988609    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kube-rbac-proxy"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988637    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="storage-initializer"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988642    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="storage-initializer"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988649    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kube-rbac-proxy"
Sep 17 18:43:00.988684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988654    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kube-rbac-proxy"
Sep 17 18:43:00.989148 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988715    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="agent"
Sep 17 18:43:00.989148 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988724    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kube-rbac-proxy"
Sep 17 18:43:00.989148 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988730    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kserve-container"
Sep 17 18:43:00.989148 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988735    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="430119a4-c1d1-4ac1-b60e-f67a7ba8c77c" containerName="kube-rbac-proxy"
Sep 17 18:43:00.989148 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.988741    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerName="kserve-container"
Sep 17 18:43:00.992332 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.992308    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:00.994459 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.994434    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:43:00.994591 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:00.994562    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 18:43:01.005084 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.005058    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"]
Sep 17 18:43:01.057877 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.057839    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"]
Sep 17 18:43:01.066425 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.064931    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.068443 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.068011    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:43:01.074047 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.074014    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"]
Sep 17 18:43:01.075949 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.075916    2582 generic.go:356] "Generic (PLEG): container finished" podID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" containerID="58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059" exitCode=0
Sep 17 18:43:01.076975 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.076945    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"
Sep 17 18:43:01.077404 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.076105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" event={"ID":"3f7dfe44-51d5-4305-b7c2-a5414a185b3b","Type":"ContainerDied","Data":"58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059"}
Sep 17 18:43:01.077505 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.077470    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg" event={"ID":"3f7dfe44-51d5-4305-b7c2-a5414a185b3b","Type":"ContainerDied","Data":"87a4b3883f7b89a6347acea82515e1c5dcaae6cac5d786e334b986fe62a909f0"}
Sep 17 18:43:01.077565 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.077538    2582 scope.go:117] "RemoveContainer" containerID="270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e"
Sep 17 18:43:01.089542 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.089378    2582 scope.go:117] "RemoveContainer" containerID="58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059"
Sep 17 18:43:01.091626 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.091599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.091840 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.091815    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7cb41c-6110-4494-a7e0-ef1937919ecd-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.091981 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.091855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf2kp\" (UniqueName: \"kubernetes.io/projected/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-api-access-sf2kp\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.092037 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.092008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.092084 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.092049    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.099757 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.099732    2582 scope.go:117] "RemoveContainer" containerID="a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5"
Sep 17 18:43:01.107033 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.107004    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"]
Sep 17 18:43:01.111193 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.111170    2582 scope.go:117] "RemoveContainer" containerID="270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e"
Sep 17 18:43:01.111489 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.111465    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7679bbb844-thdfg"]
Sep 17 18:43:01.111573 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:43:01.111550    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e\": container with ID starting with 270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e not found: ID does not exist" containerID="270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e"
Sep 17 18:43:01.111631 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.111584    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e"} err="failed to get container status \"270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e\": rpc error: code = NotFound desc = could not find container \"270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e\": container with ID starting with 270583a5270cd4524cedc47739641686837e9d27d00bd2f6d49c95a505865e9e not found: ID does not exist"
Sep 17 18:43:01.111631 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.111612    2582 scope.go:117] "RemoveContainer" containerID="58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059"
Sep 17 18:43:01.111980 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:43:01.111961    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059\": container with ID starting with 58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059 not found: ID does not exist" containerID="58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059"
Sep 17 18:43:01.112059 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.111985    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059"} err="failed to get container status \"58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059\": rpc error: code = NotFound desc = could not find container \"58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059\": container with ID starting with 58f552f888e5bae5d5942bf60da7f9e37c98f2cbec114ea472f296e5a9c1b059 not found: ID does not exist"
Sep 17 18:43:01.112059 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.112001    2582 scope.go:117] "RemoveContainer" containerID="a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5"
Sep 17 18:43:01.112292 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:43:01.112271    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5\": container with ID starting with a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5 not found: ID does not exist" containerID="a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5"
Sep 17 18:43:01.112355 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.112299    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5"} err="failed to get container status \"a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5\": rpc error: code = NotFound desc = could not find container \"a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5\": container with ID starting with a60c1adac11d7fc1858ccf06801b1c794169828cdb1d910e417402ba667ceac5 not found: ID does not exist"
Sep 17 18:43:01.192936 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.192896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f56138e7-64f0-4468-a7e9-079a91cdeaa5-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.193142 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.192953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.193142 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.192987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.193142 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.193089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7cb41c-6110-4494-a7e0-ef1937919ecd-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.193142 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.193124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf2kp\" (UniqueName: \"kubernetes.io/projected/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-api-access-sf2kp\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.193378 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.193189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.193378 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.193215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdvv\" (UniqueName: \"kubernetes.io/projected/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-api-access-5sdvv\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.193378 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.193241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.193378 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.193267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.193378 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:43:01.193271    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 18:43:01.193378 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.193296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.193378 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:43:01.193350    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7cb41c-6110-4494-a7e0-ef1937919ecd-proxy-tls podName:7f7cb41c-6110-4494-a7e0-ef1937919ecd nodeName:}" failed. No retries permitted until 2026-09-17 18:43:01.693328137 +0000 UTC m=+815.617110781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f7cb41c-6110-4494-a7e0-ef1937919ecd-proxy-tls") pod "isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" (UID: "7f7cb41c-6110-4494-a7e0-ef1937919ecd") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 18:43:01.193737 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.193706    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.193826 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.193804    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.194028 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.194012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.202778 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.202753    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf2kp\" (UniqueName: \"kubernetes.io/projected/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-api-access-sf2kp\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.294714 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.294680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.294714 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.294721    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdvv\" (UniqueName: \"kubernetes.io/projected/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-api-access-5sdvv\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.295009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.294770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.295009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.294787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f56138e7-64f0-4468-a7e9-079a91cdeaa5-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.295009 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.294807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.295009 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:43:01.294958    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:43:01.295207 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:43:01.295046    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f56138e7-64f0-4468-a7e9-079a91cdeaa5-proxy-tls podName:f56138e7-64f0-4468-a7e9-079a91cdeaa5 nodeName:}" failed. No retries permitted until 2026-09-17 18:43:01.79502539 +0000 UTC m=+815.718808033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f56138e7-64f0-4468-a7e9-079a91cdeaa5-proxy-tls") pod "isvc-canary-stop-predictor-7968cbc88d-h6r47" (UID: "f56138e7-64f0-4468-a7e9-079a91cdeaa5") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:43:01.295207 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.295175    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.295620 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.295599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.295667 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.295598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-cabundle-cert\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.309369 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.309302    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdvv\" (UniqueName: \"kubernetes.io/projected/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-api-access-5sdvv\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.697713 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.697620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7cb41c-6110-4494-a7e0-ef1937919ecd-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.700306 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.700282    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7cb41c-6110-4494-a7e0-ef1937919ecd-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.798532 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.798494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f56138e7-64f0-4468-a7e9-079a91cdeaa5-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.801073 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.801043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f56138e7-64f0-4468-a7e9-079a91cdeaa5-proxy-tls\") pod \"isvc-canary-stop-predictor-7968cbc88d-h6r47\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:01.905834 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.905791    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:01.981177 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:01.981123    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:02.182469 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:02.182435    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f7dfe44-51d5-4305-b7c2-a5414a185b3b" path="/var/lib/kubelet/pods/3f7dfe44-51d5-4305-b7c2-a5414a185b3b/volumes"
Sep 17 18:43:02.185834 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:02.185811    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"]
Sep 17 18:43:02.188687 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:43:02.188645    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7cb41c_6110_4494_a7e0_ef1937919ecd.slice/crio-5c2de9ed0ac6c39dd242fce9b84afd689e2db36cc66d368b15bdf1b1bc7696b8 WatchSource:0}: Error finding container 5c2de9ed0ac6c39dd242fce9b84afd689e2db36cc66d368b15bdf1b1bc7696b8: Status 404 returned error can't find the container with id 5c2de9ed0ac6c39dd242fce9b84afd689e2db36cc66d368b15bdf1b1bc7696b8
Sep 17 18:43:02.256633 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:02.256463    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"]
Sep 17 18:43:02.259256 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:43:02.259230    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56138e7_64f0_4468_a7e9_079a91cdeaa5.slice/crio-8787163a17d1db48ea64eb98ec8dbd27635302ae0d5a231a0572497d1811f874 WatchSource:0}: Error finding container 8787163a17d1db48ea64eb98ec8dbd27635302ae0d5a231a0572497d1811f874: Status 404 returned error can't find the container with id 8787163a17d1db48ea64eb98ec8dbd27635302ae0d5a231a0572497d1811f874
Sep 17 18:43:03.089660 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:03.089620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" event={"ID":"f56138e7-64f0-4468-a7e9-079a91cdeaa5","Type":"ContainerStarted","Data":"6af65b49158b1ac2f15f54d6f37381e5ca0665184b4cb6bdd58845cb10e845c7"}
Sep 17 18:43:03.090146 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:03.089699    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" event={"ID":"f56138e7-64f0-4468-a7e9-079a91cdeaa5","Type":"ContainerStarted","Data":"8787163a17d1db48ea64eb98ec8dbd27635302ae0d5a231a0572497d1811f874"}
Sep 17 18:43:03.091439 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:03.091414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" event={"ID":"7f7cb41c-6110-4494-a7e0-ef1937919ecd","Type":"ContainerStarted","Data":"afb602128baeccadd8eda11092720f212040ae31cc4876170670e3de9517e255"}
Sep 17 18:43:03.091548 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:03.091467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" event={"ID":"7f7cb41c-6110-4494-a7e0-ef1937919ecd","Type":"ContainerStarted","Data":"5c2de9ed0ac6c39dd242fce9b84afd689e2db36cc66d368b15bdf1b1bc7696b8"}
Sep 17 18:43:06.107679 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:06.107646    2582 generic.go:356] "Generic (PLEG): container finished" podID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerID="6af65b49158b1ac2f15f54d6f37381e5ca0665184b4cb6bdd58845cb10e845c7" exitCode=0
Sep 17 18:43:06.108130 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:06.107720    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" event={"ID":"f56138e7-64f0-4468-a7e9-079a91cdeaa5","Type":"ContainerDied","Data":"6af65b49158b1ac2f15f54d6f37381e5ca0665184b4cb6bdd58845cb10e845c7"}
Sep 17 18:43:06.116125 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:06.116093    2582 generic.go:356] "Generic (PLEG): container finished" podID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerID="afb602128baeccadd8eda11092720f212040ae31cc4876170670e3de9517e255" exitCode=0
Sep 17 18:43:06.116224 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:06.116162    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" event={"ID":"7f7cb41c-6110-4494-a7e0-ef1937919ecd","Type":"ContainerDied","Data":"afb602128baeccadd8eda11092720f212040ae31cc4876170670e3de9517e255"}
Sep 17 18:43:07.123189 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:07.123144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" event={"ID":"f56138e7-64f0-4468-a7e9-079a91cdeaa5","Type":"ContainerStarted","Data":"0161d88b0124d63fa1964cc679170b229324c850f860fdc9a6e84a16376eb37c"}
Sep 17 18:43:07.123636 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:07.123220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" event={"ID":"f56138e7-64f0-4468-a7e9-079a91cdeaa5","Type":"ContainerStarted","Data":"68e68f9f2d5b33e70cd5df0825ed0a3ca7d5da21a8228ac781d1715a0fe9f39c"}
Sep 17 18:43:07.123636 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:07.123528    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:07.123636 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:07.123585    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:07.125471 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:07.125437    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:43:07.125725 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:07.125706    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" event={"ID":"7f7cb41c-6110-4494-a7e0-ef1937919ecd","Type":"ContainerStarted","Data":"5787e3bfa73e2865ac1dbc99323da0b0533cd7ccb09d2d65691826f61e1a2578"}
Sep 17 18:43:07.125775 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:07.125760    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" event={"ID":"7f7cb41c-6110-4494-a7e0-ef1937919ecd","Type":"ContainerStarted","Data":"6a240bed49d8fbf372a8229b7acd311b1cc42cf54edad557b54cdb3e0c0385e5"}
Sep 17 18:43:07.126016 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:07.126001    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:07.175552 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:07.175501    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podStartSLOduration=2.431430485 podStartE2EDuration="6.17548288s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.744052395s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:07.15251591 +0000 UTC m=+821.076298583" watchObservedRunningTime="2026-09-17 18:43:07.17548288 +0000 UTC m=+821.099265540"
Sep 17 18:43:07.176446 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:07.176410    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podStartSLOduration=3.480283312 podStartE2EDuration="7.176396427s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.696113115s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:07.174326871 +0000 UTC m=+821.098109568" watchObservedRunningTime="2026-09-17 18:43:07.176396427 +0000 UTC m=+821.100179087"
Sep 17 18:43:08.130920 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:08.130856    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:43:08.131386 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:08.130918    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:08.132459 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:08.132430    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:43:09.135772 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:09.135726    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:43:13.135459 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:13.135429    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:43:13.135971 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:13.135946    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:43:14.140407 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:14.140378    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:43:14.140898 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:14.140856    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:43:23.136239 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:23.136190    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:43:24.141644 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:24.141598    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:43:33.135935 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:33.135896    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:43:34.141589 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:34.141545    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:43:43.135984 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:43.135934    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:43:44.141696 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:44.141654    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:43:53.136540 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:53.136498    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:43:54.141342 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:43:54.141300    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:44:03.136096 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:03.136054    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:44:04.141262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:04.141223    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:44:13.136063 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:13.135972    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:44:14.141871 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:14.141825    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:44:23.136594 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:23.136555    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:44:24.142533 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:24.142503    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:44:30.936114 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:30.936078    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"]
Sep 17 18:44:30.936555 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:30.936370    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container" containerID="cri-o://6a240bed49d8fbf372a8229b7acd311b1cc42cf54edad557b54cdb3e0c0385e5" gracePeriod=30
Sep 17 18:44:30.936555 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:30.936388    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kube-rbac-proxy" containerID="cri-o://5787e3bfa73e2865ac1dbc99323da0b0533cd7ccb09d2d65691826f61e1a2578" gracePeriod=30
Sep 17 18:44:30.972164 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:30.972129    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"]
Sep 17 18:44:30.972452 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:30.972389    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container" containerID="cri-o://68e68f9f2d5b33e70cd5df0825ed0a3ca7d5da21a8228ac781d1715a0fe9f39c" gracePeriod=30
Sep 17 18:44:30.972681 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:30.972493    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kube-rbac-proxy" containerID="cri-o://0161d88b0124d63fa1964cc679170b229324c850f860fdc9a6e84a16376eb37c" gracePeriod=30
Sep 17 18:44:31.534317 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:31.534280    2582 generic.go:356] "Generic (PLEG): container finished" podID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerID="0161d88b0124d63fa1964cc679170b229324c850f860fdc9a6e84a16376eb37c" exitCode=2
Sep 17 18:44:31.534495 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:31.534357    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" event={"ID":"f56138e7-64f0-4468-a7e9-079a91cdeaa5","Type":"ContainerDied","Data":"0161d88b0124d63fa1964cc679170b229324c850f860fdc9a6e84a16376eb37c"}
Sep 17 18:44:31.536434 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:31.536412    2582 generic.go:356] "Generic (PLEG): container finished" podID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerID="5787e3bfa73e2865ac1dbc99323da0b0533cd7ccb09d2d65691826f61e1a2578" exitCode=2
Sep 17 18:44:31.536545 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:31.536460    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" event={"ID":"7f7cb41c-6110-4494-a7e0-ef1937919ecd","Type":"ContainerDied","Data":"5787e3bfa73e2865ac1dbc99323da0b0533cd7ccb09d2d65691826f61e1a2578"}
Sep 17 18:44:33.131341 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:33.131291    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 18:44:33.136668 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:33.136637    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:44:34.136989 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:34.136923    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 18:44:34.141283 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:34.141251    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:44:36.566372 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.566331    2582 generic.go:356] "Generic (PLEG): container finished" podID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerID="68e68f9f2d5b33e70cd5df0825ed0a3ca7d5da21a8228ac781d1715a0fe9f39c" exitCode=0
Sep 17 18:44:36.566867 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.566406    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" event={"ID":"f56138e7-64f0-4468-a7e9-079a91cdeaa5","Type":"ContainerDied","Data":"68e68f9f2d5b33e70cd5df0825ed0a3ca7d5da21a8228ac781d1715a0fe9f39c"}
Sep 17 18:44:36.569354 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.569313    2582 generic.go:356] "Generic (PLEG): container finished" podID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerID="6a240bed49d8fbf372a8229b7acd311b1cc42cf54edad557b54cdb3e0c0385e5" exitCode=0
Sep 17 18:44:36.569514 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.569352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" event={"ID":"7f7cb41c-6110-4494-a7e0-ef1937919ecd","Type":"ContainerDied","Data":"6a240bed49d8fbf372a8229b7acd311b1cc42cf54edad557b54cdb3e0c0385e5"}
Sep 17 18:44:36.617961 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.617926    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:44:36.674444 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.674420    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sdvv\" (UniqueName: \"kubernetes.io/projected/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-api-access-5sdvv\") pod \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") "
Sep 17 18:44:36.674582 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.674498    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f56138e7-64f0-4468-a7e9-079a91cdeaa5-proxy-tls\") pod \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") "
Sep 17 18:44:36.674582 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.674534    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-rbac-proxy-sar-config\") pod \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") "
Sep 17 18:44:36.674582 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.674555    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kserve-provision-location\") pod \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") "
Sep 17 18:44:36.674582 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.674571    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-cabundle-cert\") pod \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\" (UID: \"f56138e7-64f0-4468-a7e9-079a91cdeaa5\") "
Sep 17 18:44:36.675113 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.674912    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f56138e7-64f0-4468-a7e9-079a91cdeaa5" (UID: "f56138e7-64f0-4468-a7e9-079a91cdeaa5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:44:36.675113 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.674872    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f56138e7-64f0-4468-a7e9-079a91cdeaa5" (UID: "f56138e7-64f0-4468-a7e9-079a91cdeaa5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:36.675113 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.674952    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f56138e7-64f0-4468-a7e9-079a91cdeaa5" (UID: "f56138e7-64f0-4468-a7e9-079a91cdeaa5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:44:36.676697 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.676664    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-api-access-5sdvv" (OuterVolumeSpecName: "kube-api-access-5sdvv") pod "f56138e7-64f0-4468-a7e9-079a91cdeaa5" (UID: "f56138e7-64f0-4468-a7e9-079a91cdeaa5"). InnerVolumeSpecName "kube-api-access-5sdvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:36.676832 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.676816    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f56138e7-64f0-4468-a7e9-079a91cdeaa5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f56138e7-64f0-4468-a7e9-079a91cdeaa5" (UID: "f56138e7-64f0-4468-a7e9-079a91cdeaa5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:36.685870 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.685848    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:44:36.775323 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775288    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-cabundle-cert\") pod \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") "
Sep 17 18:44:36.775515 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775334    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf2kp\" (UniqueName: \"kubernetes.io/projected/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-api-access-sf2kp\") pod \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") "
Sep 17 18:44:36.775515 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775370    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kserve-provision-location\") pod \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") "
Sep 17 18:44:36.775515 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775420    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-rbac-proxy-sar-config\") pod \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") "
Sep 17 18:44:36.775515 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775442    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7cb41c-6110-4494-a7e0-ef1937919ecd-proxy-tls\") pod \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\" (UID: \"7f7cb41c-6110-4494-a7e0-ef1937919ecd\") "
Sep 17 18:44:36.775725 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775638    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f56138e7-64f0-4468-a7e9-079a91cdeaa5-proxy-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:44:36.775725 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775657    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:44:36.775725 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775674    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kserve-provision-location\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:44:36.775725 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775678    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f7cb41c-6110-4494-a7e0-ef1937919ecd" (UID: "7f7cb41c-6110-4494-a7e0-ef1937919ecd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:36.775725 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775683    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f7cb41c-6110-4494-a7e0-ef1937919ecd" (UID: "7f7cb41c-6110-4494-a7e0-ef1937919ecd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:44:36.775725 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775688    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f56138e7-64f0-4468-a7e9-079a91cdeaa5-cabundle-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:44:36.775972 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775735    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sdvv\" (UniqueName: \"kubernetes.io/projected/f56138e7-64f0-4468-a7e9-079a91cdeaa5-kube-api-access-5sdvv\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:44:36.775972 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.775748    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f7cb41c-6110-4494-a7e0-ef1937919ecd" (UID: "7f7cb41c-6110-4494-a7e0-ef1937919ecd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:44:36.777764 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.777742    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-api-access-sf2kp" (OuterVolumeSpecName: "kube-api-access-sf2kp") pod "7f7cb41c-6110-4494-a7e0-ef1937919ecd" (UID: "7f7cb41c-6110-4494-a7e0-ef1937919ecd"). InnerVolumeSpecName "kube-api-access-sf2kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:36.777764 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.777751    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7cb41c-6110-4494-a7e0-ef1937919ecd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f7cb41c-6110-4494-a7e0-ef1937919ecd" (UID: "7f7cb41c-6110-4494-a7e0-ef1937919ecd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:36.876300 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.876266    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf2kp\" (UniqueName: \"kubernetes.io/projected/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-api-access-sf2kp\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:44:36.876300 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.876298    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kserve-provision-location\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:44:36.876300 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.876308    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:44:36.876513 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.876317    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f7cb41c-6110-4494-a7e0-ef1937919ecd-proxy-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:44:36.876513 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:36.876327    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f7cb41c-6110-4494-a7e0-ef1937919ecd-cabundle-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:44:37.576063 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.576028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47" event={"ID":"f56138e7-64f0-4468-a7e9-079a91cdeaa5","Type":"ContainerDied","Data":"8787163a17d1db48ea64eb98ec8dbd27635302ae0d5a231a0572497d1811f874"}
Sep 17 18:44:37.576500 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.576042    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"
Sep 17 18:44:37.576500 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.576102    2582 scope.go:117] "RemoveContainer" containerID="0161d88b0124d63fa1964cc679170b229324c850f860fdc9a6e84a16376eb37c"
Sep 17 18:44:37.578229 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.578203    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv" event={"ID":"7f7cb41c-6110-4494-a7e0-ef1937919ecd","Type":"ContainerDied","Data":"5c2de9ed0ac6c39dd242fce9b84afd689e2db36cc66d368b15bdf1b1bc7696b8"}
Sep 17 18:44:37.578318 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.578236    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"
Sep 17 18:44:37.587323 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.587305    2582 scope.go:117] "RemoveContainer" containerID="68e68f9f2d5b33e70cd5df0825ed0a3ca7d5da21a8228ac781d1715a0fe9f39c"
Sep 17 18:44:37.595739 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.595722    2582 scope.go:117] "RemoveContainer" containerID="6af65b49158b1ac2f15f54d6f37381e5ca0665184b4cb6bdd58845cb10e845c7"
Sep 17 18:44:37.604597 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.604574    2582 scope.go:117] "RemoveContainer" containerID="5787e3bfa73e2865ac1dbc99323da0b0533cd7ccb09d2d65691826f61e1a2578"
Sep 17 18:44:37.605378 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.605352    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"]
Sep 17 18:44:37.609232 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.609207    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5b79b84f69-zbrkv"]
Sep 17 18:44:37.613171 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.613153    2582 scope.go:117] "RemoveContainer" containerID="6a240bed49d8fbf372a8229b7acd311b1cc42cf54edad557b54cdb3e0c0385e5"
Sep 17 18:44:37.620453 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.620430    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"]
Sep 17 18:44:37.621301 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.621277    2582 scope.go:117] "RemoveContainer" containerID="afb602128baeccadd8eda11092720f212040ae31cc4876170670e3de9517e255"
Sep 17 18:44:37.626294 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:37.626267    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7968cbc88d-h6r47"]
Sep 17 18:44:38.183359 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:38.183326    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" path="/var/lib/kubelet/pods/7f7cb41c-6110-4494-a7e0-ef1937919ecd/volumes"
Sep 17 18:44:38.183895 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:44:38.183866    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" path="/var/lib/kubelet/pods/f56138e7-64f0-4468-a7e9-079a91cdeaa5/volumes"
Sep 17 18:45:11.691406 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691369    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"]
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691735    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kube-rbac-proxy"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691749    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kube-rbac-proxy"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691771    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="storage-initializer"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691777    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="storage-initializer"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691784    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="storage-initializer"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691790    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="storage-initializer"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691797    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kube-rbac-proxy"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691802    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kube-rbac-proxy"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691814    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691819    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691826    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691833    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691925    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kube-rbac-proxy"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691938    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f56138e7-64f0-4468-a7e9-079a91cdeaa5" containerName="kserve-container"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691950    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kserve-container"
Sep 17 18:45:11.693978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.691956    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7cb41c-6110-4494-a7e0-ef1937919ecd" containerName="kube-rbac-proxy"
Sep 17 18:45:11.694942 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.694926    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:11.697463 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.697435    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-93846-predictor-serving-cert\""
Sep 17 18:45:11.697665 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.697638    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:45:11.698275 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.698255    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-93846-kube-rbac-proxy-sar-config\""
Sep 17 18:45:11.698401 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.698255    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:45:11.698401 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.698268    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\""
Sep 17 18:45:11.705874 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.705849    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"]
Sep 17 18:45:11.788632 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.788589    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kn28\" (UniqueName: \"kubernetes.io/projected/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-api-access-5kn28\") pod \"message-dumper-raw-93846-predictor-5b9658c57d-8x9v9\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") " pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:11.788839 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.788710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-93846-predictor-5b9658c57d-8x9v9\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") " pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:11.788839 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.788755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a1089a-5d0e-48ac-b717-ce6f9a361206-proxy-tls\") pod \"message-dumper-raw-93846-predictor-5b9658c57d-8x9v9\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") " pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:11.889363 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.889325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-93846-predictor-5b9658c57d-8x9v9\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") " pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:11.889363 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.889366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a1089a-5d0e-48ac-b717-ce6f9a361206-proxy-tls\") pod \"message-dumper-raw-93846-predictor-5b9658c57d-8x9v9\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") " pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:11.889638 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.889428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kn28\" (UniqueName: \"kubernetes.io/projected/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-api-access-5kn28\") pod \"message-dumper-raw-93846-predictor-5b9658c57d-8x9v9\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") " pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:11.890099 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.890061    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-93846-predictor-5b9658c57d-8x9v9\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") " pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:11.892150 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.892124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a1089a-5d0e-48ac-b717-ce6f9a361206-proxy-tls\") pod \"message-dumper-raw-93846-predictor-5b9658c57d-8x9v9\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") " pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:11.898824 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:11.898799    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kn28\" (UniqueName: \"kubernetes.io/projected/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-api-access-5kn28\") pod \"message-dumper-raw-93846-predictor-5b9658c57d-8x9v9\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") " pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:12.009053 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:12.008938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:12.287035 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:12.287003    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"]
Sep 17 18:45:12.288602 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:45:12.288576    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a1089a_5d0e_48ac_b717_ce6f9a361206.slice/crio-cba72d9407d1e83df29ed151e320cf32e0cc64d4f1e29eb6fd065187d6641613 WatchSource:0}: Error finding container cba72d9407d1e83df29ed151e320cf32e0cc64d4f1e29eb6fd065187d6641613: Status 404 returned error can't find the container with id cba72d9407d1e83df29ed151e320cf32e0cc64d4f1e29eb6fd065187d6641613
Sep 17 18:45:12.291415 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:12.291396    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:45:12.750775 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:12.750737    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9" event={"ID":"d5a1089a-5d0e-48ac-b717-ce6f9a361206","Type":"ContainerStarted","Data":"cba72d9407d1e83df29ed151e320cf32e0cc64d4f1e29eb6fd065187d6641613"}
Sep 17 18:45:13.758107 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:13.758073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9" event={"ID":"d5a1089a-5d0e-48ac-b717-ce6f9a361206","Type":"ContainerStarted","Data":"45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e"}
Sep 17 18:45:13.758664 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:13.758140    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9" event={"ID":"d5a1089a-5d0e-48ac-b717-ce6f9a361206","Type":"ContainerStarted","Data":"34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1"}
Sep 17 18:45:13.758664 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:13.758220    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:13.779820 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:13.779762    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9" podStartSLOduration=1.527386521 podStartE2EDuration="2.779747137s" totalImagesPullingTime="1.252360616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:13.778285655 +0000 UTC m=+947.702068316" watchObservedRunningTime="2026-09-17 18:45:13.779747137 +0000 UTC m=+947.703529793"
Sep 17 18:45:14.765314 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:14.765284    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:14.767755 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:14.767731    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:21.779833 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:21.779793    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:45:31.500755 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.500721    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"]
Sep 17 18:45:31.509071 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.509046    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.511673 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.511638    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-93846-kube-rbac-proxy-sar-config\""
Sep 17 18:45:31.511838 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.511682    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-93846-predictor-serving-cert\""
Sep 17 18:45:31.511838 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.511826    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:45:31.515251 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.515228    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:45:31.518014 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.517983    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"]
Sep 17 18:45:31.559873 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.559836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.560075 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.559906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-cabundle-cert\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.560075 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.560038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksnr8\" (UniqueName: \"kubernetes.io/projected/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-api-access-ksnr8\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.560180 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.560075    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69069d3a-f903-480a-b0c7-90d0357e8b9e-proxy-tls\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.560180 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.560106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69069d3a-f903-480a-b0c7-90d0357e8b9e-kserve-provision-location\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.661330 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.661292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksnr8\" (UniqueName: \"kubernetes.io/projected/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-api-access-ksnr8\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.661330 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.661334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69069d3a-f903-480a-b0c7-90d0357e8b9e-proxy-tls\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.661606 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.661358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69069d3a-f903-480a-b0c7-90d0357e8b9e-kserve-provision-location\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.661606 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.661390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.661606 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.661433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-cabundle-cert\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.661606 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:45:31.661477    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-serving-cert: secret "isvc-logger-raw-93846-predictor-serving-cert" not found
Sep 17 18:45:31.661606 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:45:31.661557    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69069d3a-f903-480a-b0c7-90d0357e8b9e-proxy-tls podName:69069d3a-f903-480a-b0c7-90d0357e8b9e nodeName:}" failed. No retries permitted until 2026-09-17 18:45:32.161535139 +0000 UTC m=+966.085317778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/69069d3a-f903-480a-b0c7-90d0357e8b9e-proxy-tls") pod "isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" (UID: "69069d3a-f903-480a-b0c7-90d0357e8b9e") : secret "isvc-logger-raw-93846-predictor-serving-cert" not found
Sep 17 18:45:31.661929 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.661875    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69069d3a-f903-480a-b0c7-90d0357e8b9e-kserve-provision-location\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.662144 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.662128    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.662221 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.662147    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-cabundle-cert\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:31.670671 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:31.670642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksnr8\" (UniqueName: \"kubernetes.io/projected/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-api-access-ksnr8\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:32.166201 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:32.166154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69069d3a-f903-480a-b0c7-90d0357e8b9e-proxy-tls\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:32.168742 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:32.168720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69069d3a-f903-480a-b0c7-90d0357e8b9e-proxy-tls\") pod \"isvc-logger-raw-93846-predictor-56495c9f7d-csgv9\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:32.421471 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:32.421373    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:32.703955 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:32.703927    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"]
Sep 17 18:45:32.705923 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:45:32.705896    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69069d3a_f903_480a_b0c7_90d0357e8b9e.slice/crio-c4116970e9a350c32cd198e595c81479bfda55bba75848f58525b260e0fd69e9 WatchSource:0}: Error finding container c4116970e9a350c32cd198e595c81479bfda55bba75848f58525b260e0fd69e9: Status 404 returned error can't find the container with id c4116970e9a350c32cd198e595c81479bfda55bba75848f58525b260e0fd69e9
Sep 17 18:45:32.848478 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:32.848438    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" event={"ID":"69069d3a-f903-480a-b0c7-90d0357e8b9e","Type":"ContainerStarted","Data":"b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282"}
Sep 17 18:45:32.848678 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:32.848505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" event={"ID":"69069d3a-f903-480a-b0c7-90d0357e8b9e","Type":"ContainerStarted","Data":"c4116970e9a350c32cd198e595c81479bfda55bba75848f58525b260e0fd69e9"}
Sep 17 18:45:36.870284 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:36.870193    2582 generic.go:356] "Generic (PLEG): container finished" podID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerID="b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282" exitCode=0
Sep 17 18:45:36.870284 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:36.870265    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" event={"ID":"69069d3a-f903-480a-b0c7-90d0357e8b9e","Type":"ContainerDied","Data":"b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282"}
Sep 17 18:45:37.877008 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:37.876975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" event={"ID":"69069d3a-f903-480a-b0c7-90d0357e8b9e","Type":"ContainerStarted","Data":"53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4"}
Sep 17 18:45:37.877382 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:37.877044    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" event={"ID":"69069d3a-f903-480a-b0c7-90d0357e8b9e","Type":"ContainerStarted","Data":"a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc"}
Sep 17 18:45:37.877382 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:37.877070    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" event={"ID":"69069d3a-f903-480a-b0c7-90d0357e8b9e","Type":"ContainerStarted","Data":"6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046"}
Sep 17 18:45:37.877382 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:37.877369    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:37.877525 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:37.877508    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:37.878963 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:37.878934    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:45:37.901294 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:37.901241    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podStartSLOduration=3.108382835 podStartE2EDuration="6.901225356s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.792842521s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:37.90029758 +0000 UTC m=+971.824080244" watchObservedRunningTime="2026-09-17 18:45:37.901225356 +0000 UTC m=+971.825008017"
Sep 17 18:45:38.881642 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:38.881602    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:38.882155 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:38.881790    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:45:38.883048 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:38.883016    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:45:39.886716 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:39.886666    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:45:39.887187 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:39.887140    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:45:44.891266 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:44.891234    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:45:44.892036 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:44.892004    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:45:44.892261 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:44.892233    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:45:54.891866 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:54.891823    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:45:54.892271 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:45:54.892213    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:46:04.892279 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:04.892235    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:46:04.892725 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:04.892701    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:46:14.892747 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:14.892702    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:46:14.893270 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:14.893247    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:46:24.892091 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:24.892048    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:46:24.892566 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:24.892542    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:46:34.892154 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:34.892109    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:46:34.892610 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:34.892578    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:46:44.892375 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:44.892336    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:46:44.892855 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:44.892833    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:46:47.177598 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:47.177505    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:46:47.178181 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:47.178153    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:46:57.177988 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:57.177955    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:46:57.178554 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:46:57.178041    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:47:06.518923 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:06.518874    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-93846-predictor-5b9658c57d-8x9v9_d5a1089a-5d0e-48ac-b717-ce6f9a361206/kserve-container/0.log"
Sep 17 18:47:06.750369 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:06.750335    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"]
Sep 17 18:47:06.750622 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:06.750593    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" containerID="cri-o://6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046" gracePeriod=30
Sep 17 18:47:06.750788 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:06.750652    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" containerID="cri-o://53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4" gracePeriod=30
Sep 17 18:47:06.750788 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:06.750650    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kube-rbac-proxy" containerID="cri-o://a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc" gracePeriod=30
Sep 17 18:47:06.802038 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:06.802008    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"]
Sep 17 18:47:06.802278 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:06.802255    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9" podUID="d5a1089a-5d0e-48ac-b717-ce6f9a361206" containerName="kserve-container" containerID="cri-o://34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1" gracePeriod=30
Sep 17 18:47:06.802385 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:06.802270    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9" podUID="d5a1089a-5d0e-48ac-b717-ce6f9a361206" containerName="kube-rbac-proxy" containerID="cri-o://45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e" gracePeriod=30
Sep 17 18:47:07.064983 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.064914    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:47:07.177855 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.177807    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:47:07.178239 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.178209    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:47:07.212547 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.212505    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a1089a-5d0e-48ac-b717-ce6f9a361206-proxy-tls\") pod \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") "
Sep 17 18:47:07.212743 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.212565    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kn28\" (UniqueName: \"kubernetes.io/projected/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-api-access-5kn28\") pod \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") "
Sep 17 18:47:07.212743 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.212633    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-rbac-proxy-sar-config\") pod \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\" (UID: \"d5a1089a-5d0e-48ac-b717-ce6f9a361206\") "
Sep 17 18:47:07.213073 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.213043    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5a1089a-5d0e-48ac-b717-ce6f9a361206" (UID: "d5a1089a-5d0e-48ac-b717-ce6f9a361206"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:47:07.214938 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.214873    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a1089a-5d0e-48ac-b717-ce6f9a361206-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5a1089a-5d0e-48ac-b717-ce6f9a361206" (UID: "d5a1089a-5d0e-48ac-b717-ce6f9a361206"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:47:07.215033 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.214934    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-api-access-5kn28" (OuterVolumeSpecName: "kube-api-access-5kn28") pod "d5a1089a-5d0e-48ac-b717-ce6f9a361206" (UID: "d5a1089a-5d0e-48ac-b717-ce6f9a361206"). InnerVolumeSpecName "kube-api-access-5kn28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:07.300664 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.300625    2582 generic.go:356] "Generic (PLEG): container finished" podID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerID="a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc" exitCode=2
Sep 17 18:47:07.300845 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.300701    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" event={"ID":"69069d3a-f903-480a-b0c7-90d0357e8b9e","Type":"ContainerDied","Data":"a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc"}
Sep 17 18:47:07.302127 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.302108    2582 generic.go:356] "Generic (PLEG): container finished" podID="d5a1089a-5d0e-48ac-b717-ce6f9a361206" containerID="45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e" exitCode=2
Sep 17 18:47:07.302127 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.302126    2582 generic.go:356] "Generic (PLEG): container finished" podID="d5a1089a-5d0e-48ac-b717-ce6f9a361206" containerID="34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1" exitCode=2
Sep 17 18:47:07.302282 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.302154    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9" event={"ID":"d5a1089a-5d0e-48ac-b717-ce6f9a361206","Type":"ContainerDied","Data":"45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e"}
Sep 17 18:47:07.302282 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.302171    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"
Sep 17 18:47:07.302282 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.302199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9" event={"ID":"d5a1089a-5d0e-48ac-b717-ce6f9a361206","Type":"ContainerDied","Data":"34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1"}
Sep 17 18:47:07.302282 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.302221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9" event={"ID":"d5a1089a-5d0e-48ac-b717-ce6f9a361206","Type":"ContainerDied","Data":"cba72d9407d1e83df29ed151e320cf32e0cc64d4f1e29eb6fd065187d6641613"}
Sep 17 18:47:07.302282 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.302243    2582 scope.go:117] "RemoveContainer" containerID="45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e"
Sep 17 18:47:07.311266 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.311249    2582 scope.go:117] "RemoveContainer" containerID="34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1"
Sep 17 18:47:07.313428 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.313404    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5a1089a-5d0e-48ac-b717-ce6f9a361206-proxy-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:47:07.313428 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.313426    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kn28\" (UniqueName: \"kubernetes.io/projected/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-api-access-5kn28\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:47:07.313596 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.313438    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5a1089a-5d0e-48ac-b717-ce6f9a361206-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:47:07.319709 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.319690    2582 scope.go:117] "RemoveContainer" containerID="45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e"
Sep 17 18:47:07.320101 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:47:07.320084    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e\": container with ID starting with 45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e not found: ID does not exist" containerID="45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e"
Sep 17 18:47:07.320209 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.320110    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e"} err="failed to get container status \"45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e\": rpc error: code = NotFound desc = could not find container \"45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e\": container with ID starting with 45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e not found: ID does not exist"
Sep 17 18:47:07.320209 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.320128    2582 scope.go:117] "RemoveContainer" containerID="34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1"
Sep 17 18:47:07.320414 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:47:07.320399    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1\": container with ID starting with 34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1 not found: ID does not exist" containerID="34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1"
Sep 17 18:47:07.320455 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.320417    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1"} err="failed to get container status \"34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1\": rpc error: code = NotFound desc = could not find container \"34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1\": container with ID starting with 34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1 not found: ID does not exist"
Sep 17 18:47:07.320455 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.320430    2582 scope.go:117] "RemoveContainer" containerID="45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e"
Sep 17 18:47:07.320652 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.320632    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e"} err="failed to get container status \"45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e\": rpc error: code = NotFound desc = could not find container \"45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e\": container with ID starting with 45464e5c4975dee5effdb453f8a71a339e7ea74969a2f20aa87996c87467aa9e not found: ID does not exist"
Sep 17 18:47:07.320694 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.320653    2582 scope.go:117] "RemoveContainer" containerID="34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1"
Sep 17 18:47:07.320867 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.320848    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1"} err="failed to get container status \"34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1\": rpc error: code = NotFound desc = could not find container \"34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1\": container with ID starting with 34de9bb2e3050a88c6d2abb4b3c522429d736cbe3feab2e110f8c4d0c9ccfff1 not found: ID does not exist"
Sep 17 18:47:07.323772 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.323752    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"]
Sep 17 18:47:07.331844 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:07.331809    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-93846-predictor-5b9658c57d-8x9v9"]
Sep 17 18:47:08.183628 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:08.183554    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a1089a-5d0e-48ac-b717-ce6f9a361206" path="/var/lib/kubelet/pods/d5a1089a-5d0e-48ac-b717-ce6f9a361206/volumes"
Sep 17 18:47:09.886852 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:09.886807    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 18:47:12.330011 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:12.329970    2582 generic.go:356] "Generic (PLEG): container finished" podID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerID="6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046" exitCode=0
Sep 17 18:47:12.330395 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:12.330043    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" event={"ID":"69069d3a-f903-480a-b0c7-90d0357e8b9e","Type":"ContainerDied","Data":"6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046"}
Sep 17 18:47:14.887391 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:14.887347    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 18:47:17.178248 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:17.178200    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:47:17.178665 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:17.178571    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:47:19.887714 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:19.887671    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 18:47:19.888196 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:19.887837    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:47:24.887118 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:24.887056    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 18:47:27.177698 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:27.177650    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:47:27.178177 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:27.177814    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:47:27.178177 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:27.178030    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:47:27.178177 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:27.178130    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:47:29.887407 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:29.887356    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 18:47:34.887683 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:34.887640    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 18:47:36.907298 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:36.907270    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:47:36.969903 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:36.969858    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksnr8\" (UniqueName: \"kubernetes.io/projected/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-api-access-ksnr8\") pod \"69069d3a-f903-480a-b0c7-90d0357e8b9e\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") "
Sep 17 18:47:36.969903 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:36.969906    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69069d3a-f903-480a-b0c7-90d0357e8b9e-kserve-provision-location\") pod \"69069d3a-f903-480a-b0c7-90d0357e8b9e\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") "
Sep 17 18:47:36.970115 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:36.969947    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-cabundle-cert\") pod \"69069d3a-f903-480a-b0c7-90d0357e8b9e\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") "
Sep 17 18:47:36.970115 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:36.969970    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-rbac-proxy-sar-config\") pod \"69069d3a-f903-480a-b0c7-90d0357e8b9e\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") "
Sep 17 18:47:36.970115 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:36.970046    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69069d3a-f903-480a-b0c7-90d0357e8b9e-proxy-tls\") pod \"69069d3a-f903-480a-b0c7-90d0357e8b9e\" (UID: \"69069d3a-f903-480a-b0c7-90d0357e8b9e\") "
Sep 17 18:47:36.970314 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:36.970285    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69069d3a-f903-480a-b0c7-90d0357e8b9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69069d3a-f903-480a-b0c7-90d0357e8b9e" (UID: "69069d3a-f903-480a-b0c7-90d0357e8b9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:36.970454 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:36.970324    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69069d3a-f903-480a-b0c7-90d0357e8b9e" (UID: "69069d3a-f903-480a-b0c7-90d0357e8b9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:47:36.970454 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:36.970355    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69069d3a-f903-480a-b0c7-90d0357e8b9e" (UID: "69069d3a-f903-480a-b0c7-90d0357e8b9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:47:36.972439 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:36.972410    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-api-access-ksnr8" (OuterVolumeSpecName: "kube-api-access-ksnr8") pod "69069d3a-f903-480a-b0c7-90d0357e8b9e" (UID: "69069d3a-f903-480a-b0c7-90d0357e8b9e"). InnerVolumeSpecName "kube-api-access-ksnr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:36.972439 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:36.972421    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69069d3a-f903-480a-b0c7-90d0357e8b9e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69069d3a-f903-480a-b0c7-90d0357e8b9e" (UID: "69069d3a-f903-480a-b0c7-90d0357e8b9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:47:37.071172 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.071134    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksnr8\" (UniqueName: \"kubernetes.io/projected/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-api-access-ksnr8\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:47:37.071172 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.071179    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69069d3a-f903-480a-b0c7-90d0357e8b9e-kserve-provision-location\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:47:37.071374 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.071189    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-cabundle-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:47:37.071374 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.071199    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69069d3a-f903-480a-b0c7-90d0357e8b9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:47:37.071374 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.071210    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69069d3a-f903-480a-b0c7-90d0357e8b9e-proxy-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:47:37.454256 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.454159    2582 generic.go:356] "Generic (PLEG): container finished" podID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerID="53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4" exitCode=137
Sep 17 18:47:37.454256 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.454244    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" event={"ID":"69069d3a-f903-480a-b0c7-90d0357e8b9e","Type":"ContainerDied","Data":"53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4"}
Sep 17 18:47:37.454474 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.454296    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"
Sep 17 18:47:37.454474 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.454315    2582 scope.go:117] "RemoveContainer" containerID="53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4"
Sep 17 18:47:37.454474 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.454304    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9" event={"ID":"69069d3a-f903-480a-b0c7-90d0357e8b9e","Type":"ContainerDied","Data":"c4116970e9a350c32cd198e595c81479bfda55bba75848f58525b260e0fd69e9"}
Sep 17 18:47:37.463769 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.463750    2582 scope.go:117] "RemoveContainer" containerID="a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc"
Sep 17 18:47:37.472195 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.472172    2582 scope.go:117] "RemoveContainer" containerID="6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046"
Sep 17 18:47:37.486025 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.486003    2582 scope.go:117] "RemoveContainer" containerID="b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282"
Sep 17 18:47:37.491472 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.491443    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"]
Sep 17 18:47:37.494825 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.494809    2582 scope.go:117] "RemoveContainer" containerID="53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4"
Sep 17 18:47:37.495201 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:47:37.495181    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4\": container with ID starting with 53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4 not found: ID does not exist" containerID="53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4"
Sep 17 18:47:37.495277 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.495209    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4"} err="failed to get container status \"53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4\": rpc error: code = NotFound desc = could not find container \"53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4\": container with ID starting with 53b2869493708a878c10c6d9a108a0bd8bd0f20e687d9655cf8d5dca542518e4 not found: ID does not exist"
Sep 17 18:47:37.495277 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.495227    2582 scope.go:117] "RemoveContainer" containerID="a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc"
Sep 17 18:47:37.495491 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:47:37.495471    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc\": container with ID starting with a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc not found: ID does not exist" containerID="a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc"
Sep 17 18:47:37.495529 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.495501    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc"} err="failed to get container status \"a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc\": rpc error: code = NotFound desc = could not find container \"a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc\": container with ID starting with a36e68cc0dc8524ccfebeac0ba075e27e9f80b3b83d6e4177571f62109b15acc not found: ID does not exist"
Sep 17 18:47:37.495529 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.495522    2582 scope.go:117] "RemoveContainer" containerID="6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046"
Sep 17 18:47:37.495762 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:47:37.495746    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046\": container with ID starting with 6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046 not found: ID does not exist" containerID="6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046"
Sep 17 18:47:37.495824 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.495766    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046"} err="failed to get container status \"6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046\": rpc error: code = NotFound desc = could not find container \"6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046\": container with ID starting with 6b07fe80d41cc3e1effc47021c87730fb81ddbe4b6b994487873baee3af9d046 not found: ID does not exist"
Sep 17 18:47:37.495824 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.495786    2582 scope.go:117] "RemoveContainer" containerID="b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282"
Sep 17 18:47:37.496027 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:47:37.496012    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282\": container with ID starting with b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282 not found: ID does not exist" containerID="b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282"
Sep 17 18:47:37.496079 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.496030    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282"} err="failed to get container status \"b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282\": rpc error: code = NotFound desc = could not find container \"b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282\": container with ID starting with b94d5f5b5dcee0cee490ebc790fd31c7d321ef7f03d390de46ba7419a4ff6282 not found: ID does not exist"
Sep 17 18:47:37.504383 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:37.504357    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-93846-predictor-56495c9f7d-csgv9"]
Sep 17 18:47:38.182272 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:47:38.182238    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" path="/var/lib/kubelet/pods/69069d3a-f903-480a-b0c7-90d0357e8b9e/volumes"
Sep 17 18:50:02.235798 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.235746    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"]
Sep 17 18:50:02.236336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236265    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container"
Sep 17 18:50:02.236336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236284    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container"
Sep 17 18:50:02.236336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236303    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent"
Sep 17 18:50:02.236336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236312    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent"
Sep 17 18:50:02.236336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236326    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a1089a-5d0e-48ac-b717-ce6f9a361206" containerName="kserve-container"
Sep 17 18:50:02.236336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236333    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a1089a-5d0e-48ac-b717-ce6f9a361206" containerName="kserve-container"
Sep 17 18:50:02.236653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236349    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="storage-initializer"
Sep 17 18:50:02.236653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236357    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="storage-initializer"
Sep 17 18:50:02.236653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236376    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a1089a-5d0e-48ac-b717-ce6f9a361206" containerName="kube-rbac-proxy"
Sep 17 18:50:02.236653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236387    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a1089a-5d0e-48ac-b717-ce6f9a361206" containerName="kube-rbac-proxy"
Sep 17 18:50:02.236653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236399    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kube-rbac-proxy"
Sep 17 18:50:02.236653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236408    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kube-rbac-proxy"
Sep 17 18:50:02.236653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236490    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a1089a-5d0e-48ac-b717-ce6f9a361206" containerName="kserve-container"
Sep 17 18:50:02.236653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236501    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kube-rbac-proxy"
Sep 17 18:50:02.236653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236512    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a1089a-5d0e-48ac-b717-ce6f9a361206" containerName="kube-rbac-proxy"
Sep 17 18:50:02.236653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236522    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="agent"
Sep 17 18:50:02.236653 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.236534    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="69069d3a-f903-480a-b0c7-90d0357e8b9e" containerName="kserve-container"
Sep 17 18:50:02.239939 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.239918    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.243064 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.243039    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:50:02.245196 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.245136    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:50:02.245369 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.245346    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:50:02.245446 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.245348    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\""
Sep 17 18:50:02.245492 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.245358    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:50:02.245492 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.245472    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:50:02.245565 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.245360    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:50:02.255239 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.255216    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"]
Sep 17 18:50:02.290447 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.290414    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.290615 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.290512    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.290615 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.290572    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlx2\" (UniqueName: \"kubernetes.io/projected/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-api-access-ndlx2\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.290704 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.290624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.290704 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.290662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.391920 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.391867    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.392129 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.391947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.392129 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.391985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlx2\" (UniqueName: \"kubernetes.io/projected/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-api-access-ndlx2\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.392129 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:50:02.392010    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:50:02.392129 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.392016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.392129 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:50:02.392101    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-proxy-tls podName:5a7b03b8-f35f-4191-87c5-be7dc572b9d6 nodeName:}" failed. No retries permitted until 2026-09-17 18:50:02.892078309 +0000 UTC m=+1236.815860952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-proxy-tls") pod "isvc-canary-create-predictor-66d699644d-jdhsn" (UID: "5a7b03b8-f35f-4191-87c5-be7dc572b9d6") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:50:02.392421 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.392137    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.392421 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.392378    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.392936 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.392912    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.392936 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.392931    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-cabundle-cert\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.400814 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.400784    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlx2\" (UniqueName: \"kubernetes.io/projected/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-api-access-ndlx2\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.897021 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.896986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:02.899665 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:02.899637    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-proxy-tls\") pod \"isvc-canary-create-predictor-66d699644d-jdhsn\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:03.151164 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:03.151064    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:03.426306 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:03.426270    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"]
Sep 17 18:50:03.428560 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:50:03.428533    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7b03b8_f35f_4191_87c5_be7dc572b9d6.slice/crio-7f7da1ea2f1d4c99dc4f5d9e7b22b42f2aa71833cf6ed70b9139276ec475e8bb WatchSource:0}: Error finding container 7f7da1ea2f1d4c99dc4f5d9e7b22b42f2aa71833cf6ed70b9139276ec475e8bb: Status 404 returned error can't find the container with id 7f7da1ea2f1d4c99dc4f5d9e7b22b42f2aa71833cf6ed70b9139276ec475e8bb
Sep 17 18:50:04.115326 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:04.115284    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" event={"ID":"5a7b03b8-f35f-4191-87c5-be7dc572b9d6","Type":"ContainerStarted","Data":"6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb"}
Sep 17 18:50:04.115584 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:04.115344    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" event={"ID":"5a7b03b8-f35f-4191-87c5-be7dc572b9d6","Type":"ContainerStarted","Data":"7f7da1ea2f1d4c99dc4f5d9e7b22b42f2aa71833cf6ed70b9139276ec475e8bb"}
Sep 17 18:50:08.139759 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:08.139684    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerID="6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb" exitCode=0
Sep 17 18:50:08.140152 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:08.139759    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" event={"ID":"5a7b03b8-f35f-4191-87c5-be7dc572b9d6","Type":"ContainerDied","Data":"6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb"}
Sep 17 18:50:09.147539 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:09.147503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" event={"ID":"5a7b03b8-f35f-4191-87c5-be7dc572b9d6","Type":"ContainerStarted","Data":"7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b"}
Sep 17 18:50:09.147999 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:09.147566    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" event={"ID":"5a7b03b8-f35f-4191-87c5-be7dc572b9d6","Type":"ContainerStarted","Data":"5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244"}
Sep 17 18:50:09.147999 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:09.147868    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:09.148073 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:09.148046    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:09.149324 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:09.149298    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:50:09.169803 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:09.169742    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podStartSLOduration=3.514369081 podStartE2EDuration="7.169724695s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.655355614s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:09.16717616 +0000 UTC m=+1243.090958820" watchObservedRunningTime="2026-09-17 18:50:09.169724695 +0000 UTC m=+1243.093507355"
Sep 17 18:50:10.153215 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:10.153167    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:50:15.157463 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:15.157431    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:50:15.159809 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:15.157915    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:50:25.158709 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:25.158664    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:50:35.158582 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:35.158538    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:50:45.158474 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:45.158429    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:50:55.158517 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:50:55.158477    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:51:05.158443 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:05.158396    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:51:15.158060 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:15.158018    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:51:19.177208 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:19.177170    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:51:24.780585 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:24.780547    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"]
Sep 17 18:51:24.781034 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:24.780849    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container" containerID="cri-o://5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244" gracePeriod=30
Sep 17 18:51:24.781034 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:24.780966    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kube-rbac-proxy" containerID="cri-o://7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b" gracePeriod=30
Sep 17 18:51:24.781972 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:24.781953    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:51:25.153547 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:25.153497    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 18:51:25.497951 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:25.497837    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerID="7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b" exitCode=2
Sep 17 18:51:25.497951 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:25.497868    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" event={"ID":"5a7b03b8-f35f-4191-87c5-be7dc572b9d6","Type":"ContainerDied","Data":"7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b"}
Sep 17 18:51:29.823184 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.823158    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:51:29.878863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.878765    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-rbac-proxy-sar-config\") pod \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") "
Sep 17 18:51:29.878863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.878811    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-cabundle-cert\") pod \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") "
Sep 17 18:51:29.878863 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.878850    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndlx2\" (UniqueName: \"kubernetes.io/projected/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-api-access-ndlx2\") pod \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") "
Sep 17 18:51:29.879173 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.878920    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kserve-provision-location\") pod \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") "
Sep 17 18:51:29.879173 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.879001    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-proxy-tls\") pod \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\" (UID: \"5a7b03b8-f35f-4191-87c5-be7dc572b9d6\") "
Sep 17 18:51:29.879278 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.879252    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a7b03b8-f35f-4191-87c5-be7dc572b9d6" (UID: "5a7b03b8-f35f-4191-87c5-be7dc572b9d6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:29.879322 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.879269    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a7b03b8-f35f-4191-87c5-be7dc572b9d6" (UID: "5a7b03b8-f35f-4191-87c5-be7dc572b9d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:51:29.879322 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.879276    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a7b03b8-f35f-4191-87c5-be7dc572b9d6" (UID: "5a7b03b8-f35f-4191-87c5-be7dc572b9d6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:51:29.881185 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.881157    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-api-access-ndlx2" (OuterVolumeSpecName: "kube-api-access-ndlx2") pod "5a7b03b8-f35f-4191-87c5-be7dc572b9d6" (UID: "5a7b03b8-f35f-4191-87c5-be7dc572b9d6"). InnerVolumeSpecName "kube-api-access-ndlx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:51:29.881281 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.881249    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a7b03b8-f35f-4191-87c5-be7dc572b9d6" (UID: "5a7b03b8-f35f-4191-87c5-be7dc572b9d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:51:29.979990 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.979951    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndlx2\" (UniqueName: \"kubernetes.io/projected/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-api-access-ndlx2\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:51:29.979990 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.979982    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kserve-provision-location\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:51:29.979990 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.979995    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-proxy-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:51:29.980232 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.980004    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:51:29.980232 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:29.980013    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a7b03b8-f35f-4191-87c5-be7dc572b9d6-cabundle-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:51:30.523263 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.523161    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerID="5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244" exitCode=0
Sep 17 18:51:30.523263 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.523202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" event={"ID":"5a7b03b8-f35f-4191-87c5-be7dc572b9d6","Type":"ContainerDied","Data":"5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244"}
Sep 17 18:51:30.523263 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.523233    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"
Sep 17 18:51:30.523263 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.523254    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn" event={"ID":"5a7b03b8-f35f-4191-87c5-be7dc572b9d6","Type":"ContainerDied","Data":"7f7da1ea2f1d4c99dc4f5d9e7b22b42f2aa71833cf6ed70b9139276ec475e8bb"}
Sep 17 18:51:30.523580 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.523281    2582 scope.go:117] "RemoveContainer" containerID="7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b"
Sep 17 18:51:30.532191 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.532169    2582 scope.go:117] "RemoveContainer" containerID="5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244"
Sep 17 18:51:30.540128 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.540102    2582 scope.go:117] "RemoveContainer" containerID="6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb"
Sep 17 18:51:30.542484 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.542466    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"]
Sep 17 18:51:30.546235 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.546211    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d699644d-jdhsn"]
Sep 17 18:51:30.549069 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.549047    2582 scope.go:117] "RemoveContainer" containerID="7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b"
Sep 17 18:51:30.549365 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:51:30.549346    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b\": container with ID starting with 7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b not found: ID does not exist" containerID="7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b"
Sep 17 18:51:30.549443 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.549378    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b"} err="failed to get container status \"7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b\": rpc error: code = NotFound desc = could not find container \"7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b\": container with ID starting with 7ee302ff66417a4cd89365db74ea524cb6eff1cda40a18f94d36d5324cbf4c9b not found: ID does not exist"
Sep 17 18:51:30.549443 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.549403    2582 scope.go:117] "RemoveContainer" containerID="5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244"
Sep 17 18:51:30.549685 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:51:30.549667    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244\": container with ID starting with 5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244 not found: ID does not exist" containerID="5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244"
Sep 17 18:51:30.549726 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.549691    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244"} err="failed to get container status \"5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244\": rpc error: code = NotFound desc = could not find container \"5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244\": container with ID starting with 5fec147c7eb48f8dcd5254d3c739d24ed91216eba6b89f742509633726d62244 not found: ID does not exist"
Sep 17 18:51:30.549726 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.549707    2582 scope.go:117] "RemoveContainer" containerID="6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb"
Sep 17 18:51:30.550001 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:51:30.549983    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb\": container with ID starting with 6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb not found: ID does not exist" containerID="6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb"
Sep 17 18:51:30.550057 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:30.550006    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb"} err="failed to get container status \"6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb\": rpc error: code = NotFound desc = could not find container \"6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb\": container with ID starting with 6c6f510b4b74c5dc0e79dedddfc90301bfc78ea2dc402edabba48a0f5f73e2cb not found: ID does not exist"
Sep 17 18:51:32.182521 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:32.182490    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" path="/var/lib/kubelet/pods/5a7b03b8-f35f-4191-87c5-be7dc572b9d6/volumes"
Sep 17 18:51:37.515992 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.515949    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"]
Sep 17 18:51:37.516375 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.516288    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="storage-initializer"
Sep 17 18:51:37.516375 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.516299    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="storage-initializer"
Sep 17 18:51:37.516375 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.516322    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kube-rbac-proxy"
Sep 17 18:51:37.516375 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.516328    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kube-rbac-proxy"
Sep 17 18:51:37.516375 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.516338    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container"
Sep 17 18:51:37.516375 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.516344    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container"
Sep 17 18:51:37.516569 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.516395    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kube-rbac-proxy"
Sep 17 18:51:37.516569 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.516405    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a7b03b8-f35f-4191-87c5-be7dc572b9d6" containerName="kserve-container"
Sep 17 18:51:37.521087 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.521067    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.523429 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.523397    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:51:37.523599 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.523399    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:51:37.523599 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.523459    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:51:37.523599 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.523459    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:51:37.523931 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.523914    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:51:37.523999 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.523922    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:51:37.523999 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.523972    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\""
Sep 17 18:51:37.531269 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.531244    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"]
Sep 17 18:51:37.650685 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.650647    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d443520-8807-412f-9ee5-bcc8c3df2965-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.650853 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.650700    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.650853 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.650750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d443520-8807-412f-9ee5-bcc8c3df2965-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.650853 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.650781    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.650853 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.650806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phm7n\" (UniqueName: \"kubernetes.io/projected/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-api-access-phm7n\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.751722 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.751685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.751722 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.751728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phm7n\" (UniqueName: \"kubernetes.io/projected/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-api-access-phm7n\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.752045 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.751806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d443520-8807-412f-9ee5-bcc8c3df2965-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.752045 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.751857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.752045 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.751935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d443520-8807-412f-9ee5-bcc8c3df2965-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.752267 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.752238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d443520-8807-412f-9ee5-bcc8c3df2965-kserve-provision-location\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.752535 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.752513    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.752583 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.752531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-cabundle-cert\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.754495 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.754474    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d443520-8807-412f-9ee5-bcc8c3df2965-proxy-tls\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.759947 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.759928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phm7n\" (UniqueName: \"kubernetes.io/projected/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-api-access-phm7n\") pod \"isvc-canary-promote-predictor-dfdfd784d-6mxvj\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:37.833688 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:37.833654    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:38.106572 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:38.106546    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"]
Sep 17 18:51:38.109113 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:51:38.109068    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d443520_8807_412f_9ee5_bcc8c3df2965.slice/crio-3fa31755ec9617bf45381211a0ae82853e4e454e4c527d62ea0fcf4303ed5d0a WatchSource:0}: Error finding container 3fa31755ec9617bf45381211a0ae82853e4e454e4c527d62ea0fcf4303ed5d0a: Status 404 returned error can't find the container with id 3fa31755ec9617bf45381211a0ae82853e4e454e4c527d62ea0fcf4303ed5d0a
Sep 17 18:51:38.568652 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:38.568610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" event={"ID":"7d443520-8807-412f-9ee5-bcc8c3df2965","Type":"ContainerStarted","Data":"846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54"}
Sep 17 18:51:38.569098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:38.568720    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" event={"ID":"7d443520-8807-412f-9ee5-bcc8c3df2965","Type":"ContainerStarted","Data":"3fa31755ec9617bf45381211a0ae82853e4e454e4c527d62ea0fcf4303ed5d0a"}
Sep 17 18:51:42.587571 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:42.587536    2582 generic.go:356] "Generic (PLEG): container finished" podID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerID="846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54" exitCode=0
Sep 17 18:51:42.587982 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:42.587610    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" event={"ID":"7d443520-8807-412f-9ee5-bcc8c3df2965","Type":"ContainerDied","Data":"846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54"}
Sep 17 18:51:43.594014 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:43.593977    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" event={"ID":"7d443520-8807-412f-9ee5-bcc8c3df2965","Type":"ContainerStarted","Data":"206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734"}
Sep 17 18:51:43.594429 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:43.594043    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" event={"ID":"7d443520-8807-412f-9ee5-bcc8c3df2965","Type":"ContainerStarted","Data":"68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b"}
Sep 17 18:51:43.594429 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:43.594231    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:43.625737 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:43.625677    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podStartSLOduration=2.594444453 podStartE2EDuration="6.625659332s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.031214879s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:43.623187096 +0000 UTC m=+1337.546969770" watchObservedRunningTime="2026-09-17 18:51:43.625659332 +0000 UTC m=+1337.549442027"
Sep 17 18:51:44.598800 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:44.598760    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:44.600244 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:44.600216    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:51:45.603172 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:45.603129    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:51:50.608026 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:50.607992    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:51:50.608684 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:51:50.608655    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:52:00.609433 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:52:00.609389    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:52:10.608972 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:52:10.608869    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:52:20.608728 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:52:20.608683    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:52:30.608971 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:52:30.608907    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:52:40.608704 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:52:40.608648    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:52:50.609336 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:52:50.609287    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:53:00.155764 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:00.155721    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"]
Sep 17 18:53:00.156277 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:00.156018    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container" containerID="cri-o://68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b" gracePeriod=30
Sep 17 18:53:00.156277 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:00.156188    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kube-rbac-proxy" containerID="cri-o://206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734" gracePeriod=30
Sep 17 18:53:00.157185 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:00.157130    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:53:00.603913 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:00.603832    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 18:53:00.957417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:00.957330    2582 generic.go:356] "Generic (PLEG): container finished" podID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerID="206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734" exitCode=2
Sep 17 18:53:00.957417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:00.957397    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" event={"ID":"7d443520-8807-412f-9ee5-bcc8c3df2965","Type":"ContainerDied","Data":"206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734"}
Sep 17 18:53:05.210584 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.210557    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:53:05.301839 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.301803    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-cabundle-cert\") pod \"7d443520-8807-412f-9ee5-bcc8c3df2965\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") "
Sep 17 18:53:05.301839 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.301839    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-rbac-proxy-sar-config\") pod \"7d443520-8807-412f-9ee5-bcc8c3df2965\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") "
Sep 17 18:53:05.302090 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.301865    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phm7n\" (UniqueName: \"kubernetes.io/projected/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-api-access-phm7n\") pod \"7d443520-8807-412f-9ee5-bcc8c3df2965\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") "
Sep 17 18:53:05.302090 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.301922    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d443520-8807-412f-9ee5-bcc8c3df2965-kserve-provision-location\") pod \"7d443520-8807-412f-9ee5-bcc8c3df2965\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") "
Sep 17 18:53:05.302090 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.302030    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d443520-8807-412f-9ee5-bcc8c3df2965-proxy-tls\") pod \"7d443520-8807-412f-9ee5-bcc8c3df2965\" (UID: \"7d443520-8807-412f-9ee5-bcc8c3df2965\") "
Sep 17 18:53:05.302257 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.302234    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d443520-8807-412f-9ee5-bcc8c3df2965-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d443520-8807-412f-9ee5-bcc8c3df2965" (UID: "7d443520-8807-412f-9ee5-bcc8c3df2965"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:05.302307 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.302273    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7d443520-8807-412f-9ee5-bcc8c3df2965" (UID: "7d443520-8807-412f-9ee5-bcc8c3df2965"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:53:05.302307 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.302282    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d443520-8807-412f-9ee5-bcc8c3df2965" (UID: "7d443520-8807-412f-9ee5-bcc8c3df2965"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:53:05.302379 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.302368    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-cabundle-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:53:05.302420 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.302380    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:53:05.302420 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.302389    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d443520-8807-412f-9ee5-bcc8c3df2965-kserve-provision-location\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:53:05.304280 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.304258    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d443520-8807-412f-9ee5-bcc8c3df2965-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d443520-8807-412f-9ee5-bcc8c3df2965" (UID: "7d443520-8807-412f-9ee5-bcc8c3df2965"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:05.304343 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.304310    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-api-access-phm7n" (OuterVolumeSpecName: "kube-api-access-phm7n") pod "7d443520-8807-412f-9ee5-bcc8c3df2965" (UID: "7d443520-8807-412f-9ee5-bcc8c3df2965"). InnerVolumeSpecName "kube-api-access-phm7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:05.403551 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.403508    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phm7n\" (UniqueName: \"kubernetes.io/projected/7d443520-8807-412f-9ee5-bcc8c3df2965-kube-api-access-phm7n\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:53:05.403551 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.403544    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d443520-8807-412f-9ee5-bcc8c3df2965-proxy-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:53:05.981759 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.981722    2582 generic.go:356] "Generic (PLEG): container finished" podID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerID="68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b" exitCode=0
Sep 17 18:53:05.981952 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.981800    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"
Sep 17 18:53:05.981952 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.981804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" event={"ID":"7d443520-8807-412f-9ee5-bcc8c3df2965","Type":"ContainerDied","Data":"68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b"}
Sep 17 18:53:05.981952 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.981869    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj" event={"ID":"7d443520-8807-412f-9ee5-bcc8c3df2965","Type":"ContainerDied","Data":"3fa31755ec9617bf45381211a0ae82853e4e454e4c527d62ea0fcf4303ed5d0a"}
Sep 17 18:53:05.981952 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.981911    2582 scope.go:117] "RemoveContainer" containerID="206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734"
Sep 17 18:53:05.991098 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.991079    2582 scope.go:117] "RemoveContainer" containerID="68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b"
Sep 17 18:53:05.999517 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:05.999495    2582 scope.go:117] "RemoveContainer" containerID="846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54"
Sep 17 18:53:06.003417 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:06.003387    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"]
Sep 17 18:53:06.008390 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:06.008347    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-dfdfd784d-6mxvj"]
Sep 17 18:53:06.010478 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:06.010453    2582 scope.go:117] "RemoveContainer" containerID="206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734"
Sep 17 18:53:06.011058 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:53:06.011030    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734\": container with ID starting with 206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734 not found: ID does not exist" containerID="206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734"
Sep 17 18:53:06.011159 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:06.011084    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734"} err="failed to get container status \"206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734\": rpc error: code = NotFound desc = could not find container \"206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734\": container with ID starting with 206ab0cea8e02c1e68b7de26b7c94a1660a9511fceea7795ad9bff948892c734 not found: ID does not exist"
Sep 17 18:53:06.011159 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:06.011110    2582 scope.go:117] "RemoveContainer" containerID="68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b"
Sep 17 18:53:06.011501 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:53:06.011480    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b\": container with ID starting with 68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b not found: ID does not exist" containerID="68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b"
Sep 17 18:53:06.011578 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:06.011507    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b"} err="failed to get container status \"68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b\": rpc error: code = NotFound desc = could not find container \"68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b\": container with ID starting with 68983b1f2a8633e7c629be4dab3b84aa13ef56aab3df8b130b9a55db84a6ed7b not found: ID does not exist"
Sep 17 18:53:06.011578 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:06.011526    2582 scope.go:117] "RemoveContainer" containerID="846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54"
Sep 17 18:53:06.011861 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:53:06.011842    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54\": container with ID starting with 846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54 not found: ID does not exist" containerID="846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54"
Sep 17 18:53:06.011942 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:06.011878    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54"} err="failed to get container status \"846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54\": rpc error: code = NotFound desc = could not find container \"846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54\": container with ID starting with 846570d7d4a71f8cd2755b17bc1d4c99b9ba280c3ea2cf61033daafbe8a24b54 not found: ID does not exist"
Sep 17 18:53:06.182332 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:06.182297    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" path="/var/lib/kubelet/pods/7d443520-8807-412f-9ee5-bcc8c3df2965/volumes"
Sep 17 18:53:07.653981 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.653945    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"]
Sep 17 18:53:07.654354 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.654292    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="storage-initializer"
Sep 17 18:53:07.654354 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.654304    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="storage-initializer"
Sep 17 18:53:07.654354 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.654322    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container"
Sep 17 18:53:07.654354 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.654328    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container"
Sep 17 18:53:07.654354 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.654337    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kube-rbac-proxy"
Sep 17 18:53:07.654354 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.654343    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kube-rbac-proxy"
Sep 17 18:53:07.654555 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.654398    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kube-rbac-proxy"
Sep 17 18:53:07.654555 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.654406    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d443520-8807-412f-9ee5-bcc8c3df2965" containerName="kserve-container"
Sep 17 18:53:07.659226 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.659204    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.661998 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.661976    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:53:07.662857 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.662836    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:53:07.663006 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.662980    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:53:07.663006 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.663000    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\""
Sep 17 18:53:07.663136 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.663000    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:53:07.663136 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.662985    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:53:07.663346 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.663330    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:53:07.669864 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.669844    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"]
Sep 17 18:53:07.719477 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.719437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.719671 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.719487    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzsn7\" (UniqueName: \"kubernetes.io/projected/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-api-access-jzsn7\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.719671 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.719549    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.719671 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.719598    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6334d4b4-9687-4fc1-9085-b6dddf78646c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.719671 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.719623    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6334d4b4-9687-4fc1-9085-b6dddf78646c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.820903 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.820854    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.821102 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.820932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzsn7\" (UniqueName: \"kubernetes.io/projected/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-api-access-jzsn7\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.821102 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.820965    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.821102 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.821003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6334d4b4-9687-4fc1-9085-b6dddf78646c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.821102 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.821021    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6334d4b4-9687-4fc1-9085-b6dddf78646c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.821429 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.821401    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6334d4b4-9687-4fc1-9085-b6dddf78646c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.821666 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.821645    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.821728 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.821670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.823573 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.823552    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6334d4b4-9687-4fc1-9085-b6dddf78646c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.829254 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.829233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzsn7\" (UniqueName: \"kubernetes.io/projected/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-api-access-jzsn7\") pod \"isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:07.970760 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:07.970662    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:08.247794 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:08.247768    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"]
Sep 17 18:53:08.250058 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:53:08.250021    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6334d4b4_9687_4fc1_9085_b6dddf78646c.slice/crio-5d599df0c0b620c2e4ba14d1ee496a4554ff9d5112736f6dc31f56ee0c57ae40 WatchSource:0}: Error finding container 5d599df0c0b620c2e4ba14d1ee496a4554ff9d5112736f6dc31f56ee0c57ae40: Status 404 returned error can't find the container with id 5d599df0c0b620c2e4ba14d1ee496a4554ff9d5112736f6dc31f56ee0c57ae40
Sep 17 18:53:08.998707 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:08.998669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" event={"ID":"6334d4b4-9687-4fc1-9085-b6dddf78646c","Type":"ContainerStarted","Data":"dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8"}
Sep 17 18:53:08.999132 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:08.998733    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" event={"ID":"6334d4b4-9687-4fc1-9085-b6dddf78646c","Type":"ContainerStarted","Data":"5d599df0c0b620c2e4ba14d1ee496a4554ff9d5112736f6dc31f56ee0c57ae40"}
Sep 17 18:53:13.017712 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:13.017678    2582 generic.go:356] "Generic (PLEG): container finished" podID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerID="dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8" exitCode=0
Sep 17 18:53:13.017712 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:13.017716    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" event={"ID":"6334d4b4-9687-4fc1-9085-b6dddf78646c","Type":"ContainerDied","Data":"dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8"}
Sep 17 18:53:14.025312 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:14.025274    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" event={"ID":"6334d4b4-9687-4fc1-9085-b6dddf78646c","Type":"ContainerStarted","Data":"c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92"}
Sep 17 18:53:14.025686 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:14.025340    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" event={"ID":"6334d4b4-9687-4fc1-9085-b6dddf78646c","Type":"ContainerStarted","Data":"260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41"}
Sep 17 18:53:14.025686 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:14.025526    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:14.045471 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:14.045424    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podStartSLOduration=3.331778011 podStartE2EDuration="7.045409117s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.713631106s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:53:14.043623822 +0000 UTC m=+1427.967406484" watchObservedRunningTime="2026-09-17 18:53:14.045409117 +0000 UTC m=+1427.969191780"
Sep 17 18:53:15.030640 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:15.030607    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:15.032316 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:15.032285    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:53:16.035512 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:16.035472    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:53:21.040654 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:21.040619    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:53:21.041167 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:21.041138    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:53:31.042069 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:31.042022    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:53:41.041616 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:41.041575    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:53:51.041449 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:53:51.041408    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:54:01.041930 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:01.041862    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:54:11.041836 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:11.041787    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:54:21.041446 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:21.041403    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:54:27.177125 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:27.177084    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:54:37.179037 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:37.179003    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:54:37.808173 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:37.808140    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"]
Sep 17 18:54:37.808404 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:37.808382    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container" containerID="cri-o://260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41" gracePeriod=30
Sep 17 18:54:37.808498 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:37.808455    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kube-rbac-proxy" containerID="cri-o://c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92" gracePeriod=30
Sep 17 18:54:38.427264 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:38.427229    2582 generic.go:356] "Generic (PLEG): container finished" podID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerID="c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92" exitCode=2
Sep 17 18:54:38.427636 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:38.427277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" event={"ID":"6334d4b4-9687-4fc1-9085-b6dddf78646c","Type":"ContainerDied","Data":"c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92"}
Sep 17 18:54:41.036213 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:41.036162    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 18:54:42.962732 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:42.962707    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:54:43.144930 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.144845    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6334d4b4-9687-4fc1-9085-b6dddf78646c-proxy-tls\") pod \"6334d4b4-9687-4fc1-9085-b6dddf78646c\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") "
Sep 17 18:54:43.145171 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.144970    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-rbac-proxy-sar-config\") pod \"6334d4b4-9687-4fc1-9085-b6dddf78646c\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") "
Sep 17 18:54:43.145171 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.144993    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-cabundle-cert\") pod \"6334d4b4-9687-4fc1-9085-b6dddf78646c\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") "
Sep 17 18:54:43.145171 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.145023    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6334d4b4-9687-4fc1-9085-b6dddf78646c-kserve-provision-location\") pod \"6334d4b4-9687-4fc1-9085-b6dddf78646c\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") "
Sep 17 18:54:43.145171 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.145067    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzsn7\" (UniqueName: \"kubernetes.io/projected/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-api-access-jzsn7\") pod \"6334d4b4-9687-4fc1-9085-b6dddf78646c\" (UID: \"6334d4b4-9687-4fc1-9085-b6dddf78646c\") "
Sep 17 18:54:43.145414 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.145376    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6334d4b4-9687-4fc1-9085-b6dddf78646c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6334d4b4-9687-4fc1-9085-b6dddf78646c" (UID: "6334d4b4-9687-4fc1-9085-b6dddf78646c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:43.145414 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.145390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6334d4b4-9687-4fc1-9085-b6dddf78646c" (UID: "6334d4b4-9687-4fc1-9085-b6dddf78646c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:43.145501 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.145395    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6334d4b4-9687-4fc1-9085-b6dddf78646c" (UID: "6334d4b4-9687-4fc1-9085-b6dddf78646c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:43.147164 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.147137    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6334d4b4-9687-4fc1-9085-b6dddf78646c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6334d4b4-9687-4fc1-9085-b6dddf78646c" (UID: "6334d4b4-9687-4fc1-9085-b6dddf78646c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:54:43.147281 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.147224    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-api-access-jzsn7" (OuterVolumeSpecName: "kube-api-access-jzsn7") pod "6334d4b4-9687-4fc1-9085-b6dddf78646c" (UID: "6334d4b4-9687-4fc1-9085-b6dddf78646c"). InnerVolumeSpecName "kube-api-access-jzsn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:43.246190 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.246150    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6334d4b4-9687-4fc1-9085-b6dddf78646c-proxy-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:54:43.246190 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.246182    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:54:43.246190 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.246191    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6334d4b4-9687-4fc1-9085-b6dddf78646c-cabundle-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:54:43.246190 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.246200    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6334d4b4-9687-4fc1-9085-b6dddf78646c-kserve-provision-location\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:54:43.246456 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.246209    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzsn7\" (UniqueName: \"kubernetes.io/projected/6334d4b4-9687-4fc1-9085-b6dddf78646c-kube-api-access-jzsn7\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:54:43.451168 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.451075    2582 generic.go:356] "Generic (PLEG): container finished" podID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerID="260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41" exitCode=0
Sep 17 18:54:43.451168 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.451135    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"
Sep 17 18:54:43.451168 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.451151    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" event={"ID":"6334d4b4-9687-4fc1-9085-b6dddf78646c","Type":"ContainerDied","Data":"260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41"}
Sep 17 18:54:43.451418 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.451214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd" event={"ID":"6334d4b4-9687-4fc1-9085-b6dddf78646c","Type":"ContainerDied","Data":"5d599df0c0b620c2e4ba14d1ee496a4554ff9d5112736f6dc31f56ee0c57ae40"}
Sep 17 18:54:43.451418 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.451240    2582 scope.go:117] "RemoveContainer" containerID="c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92"
Sep 17 18:54:43.460579 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.460562    2582 scope.go:117] "RemoveContainer" containerID="260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41"
Sep 17 18:54:43.468272 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.468253    2582 scope.go:117] "RemoveContainer" containerID="dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8"
Sep 17 18:54:43.472785 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.472763    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"]
Sep 17 18:54:43.476783 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.476766    2582 scope.go:117] "RemoveContainer" containerID="c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92"
Sep 17 18:54:43.477145 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:54:43.477121    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92\": container with ID starting with c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92 not found: ID does not exist" containerID="c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92"
Sep 17 18:54:43.477216 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.477159    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92"} err="failed to get container status \"c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92\": rpc error: code = NotFound desc = could not find container \"c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92\": container with ID starting with c6aa1b7e4a388386b92648aa511d32e708b1ec1938da6e232a9b57fd2694cd92 not found: ID does not exist"
Sep 17 18:54:43.477216 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.477178    2582 scope.go:117] "RemoveContainer" containerID="260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41"
Sep 17 18:54:43.477413 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:54:43.477398    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41\": container with ID starting with 260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41 not found: ID does not exist" containerID="260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41"
Sep 17 18:54:43.477453 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.477419    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41"} err="failed to get container status \"260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41\": rpc error: code = NotFound desc = could not find container \"260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41\": container with ID starting with 260cc358f04aa648df5ad048a272ef34f2c4deb37be418c532793881d0c91b41 not found: ID does not exist"
Sep 17 18:54:43.477453 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.477439    2582 scope.go:117] "RemoveContainer" containerID="dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8"
Sep 17 18:54:43.477692 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:54:43.477677    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8\": container with ID starting with dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8 not found: ID does not exist" containerID="dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8"
Sep 17 18:54:43.477729 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.477698    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8"} err="failed to get container status \"dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8\": rpc error: code = NotFound desc = could not find container \"dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8\": container with ID starting with dea9b6e00d9b612c98a6c6312afa8bb2e3e3208da81f96e55d5fb0331cd071b8 not found: ID does not exist"
Sep 17 18:54:43.482586 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:43.482564    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f7dc9cbc5-pr4nd"]
Sep 17 18:54:44.182063 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:44.182029    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" path="/var/lib/kubelet/pods/6334d4b4-9687-4fc1-9085-b6dddf78646c/volumes"
Sep 17 18:54:53.233808 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.233768    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"]
Sep 17 18:54:53.234228 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.234138    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container"
Sep 17 18:54:53.234228 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.234150    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container"
Sep 17 18:54:53.234228 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.234159    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="storage-initializer"
Sep 17 18:54:53.234228 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.234165    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="storage-initializer"
Sep 17 18:54:53.234228 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.234175    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kube-rbac-proxy"
Sep 17 18:54:53.234228 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.234181    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kube-rbac-proxy"
Sep 17 18:54:53.234414 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.234235    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kube-rbac-proxy"
Sep 17 18:54:53.234414 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.234244    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6334d4b4-9687-4fc1-9085-b6dddf78646c" containerName="kserve-container"
Sep 17 18:54:53.237410 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.237387    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:53.239857 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.239834    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 17 18:54:53.240715 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.240694    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chzzm\""
Sep 17 18:54:53.240867 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.240695    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:54:53.240867 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.240695    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:54:53.240867 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.240694    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 18:54:53.247538 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.247511    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"]
Sep 17 18:54:53.329268 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.329232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-trkvq\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:53.329268 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.329274    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-trkvq\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:53.329500 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.329312    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7854\" (UniqueName: \"kubernetes.io/projected/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-api-access-q7854\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-trkvq\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:53.430331 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.430293    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-trkvq\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:53.430331 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.430336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-trkvq\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:53.430550 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.430376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7854\" (UniqueName: \"kubernetes.io/projected/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-api-access-q7854\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-trkvq\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:53.430989 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.430971    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-trkvq\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:53.432932 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.432903    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-trkvq\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:53.440109 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.440083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7854\" (UniqueName: \"kubernetes.io/projected/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-api-access-q7854\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-trkvq\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:53.549066 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.549032    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:53.826482 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:53.826389    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"]
Sep 17 18:54:53.829365 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:54:53.829336    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8e9238_21d8_43fe_96f1_d4ddb7f31064.slice/crio-227cbfd22b26c3e602d54e2a8886094649f3369235278a6daba0ea3cd38efd03 WatchSource:0}: Error finding container 227cbfd22b26c3e602d54e2a8886094649f3369235278a6daba0ea3cd38efd03: Status 404 returned error can't find the container with id 227cbfd22b26c3e602d54e2a8886094649f3369235278a6daba0ea3cd38efd03
Sep 17 18:54:54.504362 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:54.504321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq" event={"ID":"5a8e9238-21d8-43fe-96f1-d4ddb7f31064","Type":"ContainerStarted","Data":"857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5"}
Sep 17 18:54:54.504758 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:54.504384    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq" event={"ID":"5a8e9238-21d8-43fe-96f1-d4ddb7f31064","Type":"ContainerStarted","Data":"640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a"}
Sep 17 18:54:54.504758 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:54.504404    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq" event={"ID":"5a8e9238-21d8-43fe-96f1-d4ddb7f31064","Type":"ContainerStarted","Data":"227cbfd22b26c3e602d54e2a8886094649f3369235278a6daba0ea3cd38efd03"}
Sep 17 18:54:54.504758 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:54.504484    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:54.538136 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:54.538082    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq" podStartSLOduration=1.538052729 podStartE2EDuration="1.538052729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:54.537130248 +0000 UTC m=+1528.460912908" watchObservedRunningTime="2026-09-17 18:54:54.538052729 +0000 UTC m=+1528.461835391"
Sep 17 18:54:55.512456 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:55.512424    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:54:55.514404 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:54:55.514382    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:55:02.525599 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:02.525566    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:55:03.238802 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.238761    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"]
Sep 17 18:55:03.243421 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.243396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.245911 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.245870    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:55:03.246055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.245874    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 18:55:03.246055 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.245870    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 17 18:55:03.246138 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.246084    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:55:03.255421 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.255395    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"]
Sep 17 18:55:03.318645 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.318607    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.318812 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.318649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.318812 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.318678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfdq\" (UniqueName: \"kubernetes.io/projected/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-api-access-tlfdq\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.318812 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.318724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1159cdf-1d03-47ab-b93d-e5e252940705-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.318812 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.318751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1159cdf-1d03-47ab-b93d-e5e252940705-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.420050 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.420005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.420050 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.420056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.420305 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.420120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfdq\" (UniqueName: \"kubernetes.io/projected/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-api-access-tlfdq\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.420305 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.420163    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1159cdf-1d03-47ab-b93d-e5e252940705-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.420305 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.420190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1159cdf-1d03-47ab-b93d-e5e252940705-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.420449 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:55:03.420360    2582 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-serving-cert: secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 17 18:55:03.420503 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:55:03.420461    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1159cdf-1d03-47ab-b93d-e5e252940705-proxy-tls podName:b1159cdf-1d03-47ab-b93d-e5e252940705 nodeName:}" failed. No retries permitted until 2026-09-17 18:55:03.92043845 +0000 UTC m=+1537.844221103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b1159cdf-1d03-47ab-b93d-e5e252940705-proxy-tls") pod "isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" (UID: "b1159cdf-1d03-47ab-b93d-e5e252940705") : secret "isvc-logger-raw-cipn-predictor-serving-cert" not found
Sep 17 18:55:03.420618 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.420598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1159cdf-1d03-47ab-b93d-e5e252940705-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.420785 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.420767    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.420850 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.420834    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.429262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.429232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfdq\" (UniqueName: \"kubernetes.io/projected/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-api-access-tlfdq\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.924787 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.924737    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1159cdf-1d03-47ab-b93d-e5e252940705-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:03.927464 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:03.927435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1159cdf-1d03-47ab-b93d-e5e252940705-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7dc457b548-npdh9\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:04.157473 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:04.157431    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:04.449178 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:04.449151    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"]
Sep 17 18:55:04.451841 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:55:04.451812    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1159cdf_1d03_47ab_b93d_e5e252940705.slice/crio-6a52983a8b6e9794c927780c576247a42407d603b252270d06ffe41f9bdd0a2d WatchSource:0}: Error finding container 6a52983a8b6e9794c927780c576247a42407d603b252270d06ffe41f9bdd0a2d: Status 404 returned error can't find the container with id 6a52983a8b6e9794c927780c576247a42407d603b252270d06ffe41f9bdd0a2d
Sep 17 18:55:04.554777 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:04.554742    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" event={"ID":"b1159cdf-1d03-47ab-b93d-e5e252940705","Type":"ContainerStarted","Data":"6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74"}
Sep 17 18:55:04.554943 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:04.554808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" event={"ID":"b1159cdf-1d03-47ab-b93d-e5e252940705","Type":"ContainerStarted","Data":"6a52983a8b6e9794c927780c576247a42407d603b252270d06ffe41f9bdd0a2d"}
Sep 17 18:55:08.574856 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:08.574820    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerID="6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74" exitCode=0
Sep 17 18:55:08.575254 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:08.574917    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" event={"ID":"b1159cdf-1d03-47ab-b93d-e5e252940705","Type":"ContainerDied","Data":"6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74"}
Sep 17 18:55:09.581798 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:09.581764    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" event={"ID":"b1159cdf-1d03-47ab-b93d-e5e252940705","Type":"ContainerStarted","Data":"ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239"}
Sep 17 18:55:09.582231 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:09.581843    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" event={"ID":"b1159cdf-1d03-47ab-b93d-e5e252940705","Type":"ContainerStarted","Data":"bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387"}
Sep 17 18:55:09.582231 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:09.581870    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" event={"ID":"b1159cdf-1d03-47ab-b93d-e5e252940705","Type":"ContainerStarted","Data":"4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275"}
Sep 17 18:55:09.582231 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:09.582172    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:09.582419 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:09.582333    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:09.583628 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:09.583606    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:55:09.605968 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:09.605914    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podStartSLOduration=2.796657973 podStartE2EDuration="6.605898519s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.809240546s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:09.604705372 +0000 UTC m=+1543.528488033" watchObservedRunningTime="2026-09-17 18:55:09.605898519 +0000 UTC m=+1543.529681171"
Sep 17 18:55:10.586983 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:10.586947    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:10.587414 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:10.587078    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:55:10.588358 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:10.588324    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:55:11.591146 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:11.591102    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:55:11.591730 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:11.591703    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:55:16.595921 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:16.595863    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:55:16.598260 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:16.596477    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:55:16.598260 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:16.596731    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:55:26.596562 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:26.596511    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:55:26.597071 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:26.597045    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:55:36.596665 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:36.596618    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:55:36.597189 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:36.597142    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:55:46.597049 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:46.596988    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:55:46.597536 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:46.597421    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:55:56.597042 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:56.597000    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:55:56.597571 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:55:56.597545    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:56:06.596993 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:06.596942    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:56:06.597441 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:06.597398    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:56:16.596538 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:16.596496    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:56:16.597076 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:16.597050    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:56:26.597485 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:26.597446    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:56:26.598113 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:26.598089    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:56:38.281421 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.281393    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-trkvq_5a8e9238-21d8-43fe-96f1-d4ddb7f31064/kserve-container/0.log"
Sep 17 18:56:38.477781 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.477747    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"]
Sep 17 18:56:38.478059 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.478018    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" containerID="cri-o://4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275" gracePeriod=30
Sep 17 18:56:38.478221 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.478042    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" containerID="cri-o://ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239" gracePeriod=30
Sep 17 18:56:38.478221 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.478066    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kube-rbac-proxy" containerID="cri-o://bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387" gracePeriod=30
Sep 17 18:56:38.523334 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.523294    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"]
Sep 17 18:56:38.523541 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.523514    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq" podUID="5a8e9238-21d8-43fe-96f1-d4ddb7f31064" containerName="kserve-container" containerID="cri-o://640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a" gracePeriod=30
Sep 17 18:56:38.523593 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.523564    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq" podUID="5a8e9238-21d8-43fe-96f1-d4ddb7f31064" containerName="kube-rbac-proxy" containerID="cri-o://857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5" gracePeriod=30
Sep 17 18:56:38.782119 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.782094    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:56:38.863411 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.863373    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-proxy-tls\") pod \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") "
Sep 17 18:56:38.863593 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.863466    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-rbac-proxy-sar-config\") pod \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") "
Sep 17 18:56:38.863636 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.863597    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7854\" (UniqueName: \"kubernetes.io/projected/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-api-access-q7854\") pod \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\" (UID: \"5a8e9238-21d8-43fe-96f1-d4ddb7f31064\") "
Sep 17 18:56:38.863810 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.863788    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5a8e9238-21d8-43fe-96f1-d4ddb7f31064" (UID: "5a8e9238-21d8-43fe-96f1-d4ddb7f31064"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:38.865731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.865706    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5a8e9238-21d8-43fe-96f1-d4ddb7f31064" (UID: "5a8e9238-21d8-43fe-96f1-d4ddb7f31064"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:56:38.865849 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.865750    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-api-access-q7854" (OuterVolumeSpecName: "kube-api-access-q7854") pod "5a8e9238-21d8-43fe-96f1-d4ddb7f31064" (UID: "5a8e9238-21d8-43fe-96f1-d4ddb7f31064"). InnerVolumeSpecName "kube-api-access-q7854". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:38.964976 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.964934    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7854\" (UniqueName: \"kubernetes.io/projected/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-api-access-q7854\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:56:38.964976 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.964969    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-proxy-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:56:38.964976 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:38.964978    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5a8e9238-21d8-43fe-96f1-d4ddb7f31064-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:56:39.004026 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.003986    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a8e9238-21d8-43fe-96f1-d4ddb7f31064" containerID="857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5" exitCode=2
Sep 17 18:56:39.004026 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.004017    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a8e9238-21d8-43fe-96f1-d4ddb7f31064" containerID="640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a" exitCode=2
Sep 17 18:56:39.004283 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.004041    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"
Sep 17 18:56:39.004283 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.004072    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq" event={"ID":"5a8e9238-21d8-43fe-96f1-d4ddb7f31064","Type":"ContainerDied","Data":"857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5"}
Sep 17 18:56:39.004283 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.004179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq" event={"ID":"5a8e9238-21d8-43fe-96f1-d4ddb7f31064","Type":"ContainerDied","Data":"640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a"}
Sep 17 18:56:39.004283 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.004213    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq" event={"ID":"5a8e9238-21d8-43fe-96f1-d4ddb7f31064","Type":"ContainerDied","Data":"227cbfd22b26c3e602d54e2a8886094649f3369235278a6daba0ea3cd38efd03"}
Sep 17 18:56:39.004283 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.004249    2582 scope.go:117] "RemoveContainer" containerID="857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5"
Sep 17 18:56:39.006813 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.006789    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerID="bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387" exitCode=2
Sep 17 18:56:39.006954 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.006872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" event={"ID":"b1159cdf-1d03-47ab-b93d-e5e252940705","Type":"ContainerDied","Data":"bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387"}
Sep 17 18:56:39.015350 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.015329    2582 scope.go:117] "RemoveContainer" containerID="640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a"
Sep 17 18:56:39.025410 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.025371    2582 scope.go:117] "RemoveContainer" containerID="857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5"
Sep 17 18:56:39.025727 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:56:39.025709    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5\": container with ID starting with 857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5 not found: ID does not exist" containerID="857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5"
Sep 17 18:56:39.025804 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.025735    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5"} err="failed to get container status \"857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5\": rpc error: code = NotFound desc = could not find container \"857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5\": container with ID starting with 857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5 not found: ID does not exist"
Sep 17 18:56:39.025804 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.025755    2582 scope.go:117] "RemoveContainer" containerID="640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a"
Sep 17 18:56:39.026035 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:56:39.026012    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a\": container with ID starting with 640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a not found: ID does not exist" containerID="640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a"
Sep 17 18:56:39.026094 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.026042    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a"} err="failed to get container status \"640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a\": rpc error: code = NotFound desc = could not find container \"640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a\": container with ID starting with 640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a not found: ID does not exist"
Sep 17 18:56:39.026094 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.026059    2582 scope.go:117] "RemoveContainer" containerID="857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5"
Sep 17 18:56:39.026319 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.026302    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5"} err="failed to get container status \"857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5\": rpc error: code = NotFound desc = could not find container \"857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5\": container with ID starting with 857a2dede6e0dcc00f9fd59dc3fdbb41424833829411ffc29bc227ed3c5d66c5 not found: ID does not exist"
Sep 17 18:56:39.026319 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.026318    2582 scope.go:117] "RemoveContainer" containerID="640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a"
Sep 17 18:56:39.026536 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.026520    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a"} err="failed to get container status \"640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a\": rpc error: code = NotFound desc = could not find container \"640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a\": container with ID starting with 640177c6996273e7f3052d7455eb7ff922d52b38b124a5adf3456616cbdfd18a not found: ID does not exist"
Sep 17 18:56:39.027316 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.027295    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"]
Sep 17 18:56:39.031892 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:39.031855    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-trkvq"]
Sep 17 18:56:40.183114 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:40.183079    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a8e9238-21d8-43fe-96f1-d4ddb7f31064" path="/var/lib/kubelet/pods/5a8e9238-21d8-43fe-96f1-d4ddb7f31064/volumes"
Sep 17 18:56:41.591235 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:41.591189    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 18:56:44.036029 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:44.035993    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerID="4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275" exitCode=0
Sep 17 18:56:44.036401 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:44.036061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" event={"ID":"b1159cdf-1d03-47ab-b93d-e5e252940705","Type":"ContainerDied","Data":"4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275"}
Sep 17 18:56:46.591242 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:46.591199    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 18:56:46.597332 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:46.597295    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:56:46.598121 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:46.598094    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:56:51.592183 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:51.592143    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 18:56:51.592571 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:51.592292    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:56:55.970060 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:55.970023    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:56:55.999418 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:55.999386    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:56:56.012949 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:56.012921    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:56:56.025790 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:56.025770    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:56:56.591642 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:56.591598    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 18:56:56.596786 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:56.596759    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:56:56.599030 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:56.599009    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:56:56.875434 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:56.875347    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:56:56.887006 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:56.886977    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:56:56.897133 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:56.897116    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:56:56.907989 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:56.907958    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:56:57.746388 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:57.746359    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:56:57.766539 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:57.766496    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:56:57.779449 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:57.779424    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:56:57.796301 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:57.796280    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:56:58.658570 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:58.658539    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:56:58.691087 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:58.691057    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:56:58.717130 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:58.717102    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:56:58.739309 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:58.739280    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:56:59.635097 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:59.635068    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:56:59.652596 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:59.652563    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:56:59.669564 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:59.669536    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:56:59.683578 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:56:59.683558    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:57:00.513172 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:00.513142    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:57:00.525895 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:00.525860    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:57:00.543585 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:00.543558    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:57:00.558208 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:00.558168    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:57:01.391295 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:01.391254    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:57:01.406745 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:01.406720    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:57:01.424153 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:01.424122    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:57:01.436138 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:01.436112    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:57:01.591470 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:01.591424    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 18:57:02.266000 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:02.265956    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:57:02.297573 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:02.297546    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:57:02.326080 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:02.326026    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:57:02.349831 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:02.349798    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:57:03.188772 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:03.188739    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:57:03.202569 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:03.202540    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:57:03.215254 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:03.215228    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:57:03.233987 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:03.233957    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:57:04.075474 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:04.075443    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:57:04.088281 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:04.088253    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:57:04.100101 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:04.100072    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:57:04.112705 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:04.112679    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:57:04.997109 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:04.997080    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:57:05.015501 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:05.015469    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:57:05.026487 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:05.026462    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:57:05.038215 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:05.038195    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:57:05.865082 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:05.865051    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:57:05.878211 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:05.878182    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:57:05.889834 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:05.889808    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:57:05.902312 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:05.902288    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:57:06.591436 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:06.591394    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 18:57:06.597428 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:06.597391    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:57:06.597575 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:06.597509    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:57:06.598807 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:06.598786    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:57:06.598915 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:06.598893    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:57:06.719978 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:06.719953    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:57:06.735282 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:06.735255    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:57:06.747858 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:06.747829    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:57:06.760420 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:06.760395    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:57:07.609010 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:07.608978    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:57:07.628356 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:07.623771    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:57:07.637694 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:07.637666    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:57:07.651045 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:07.651022    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:57:08.561762 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.561733    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kserve-container/0.log"
Sep 17 18:57:08.575119 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.575094    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/kube-rbac-proxy/0.log"
Sep 17 18:57:08.589550 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.589526    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/agent/0.log"
Sep 17 18:57:08.601789 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.601767    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7dc457b548-npdh9_b1159cdf-1d03-47ab-b93d-e5e252940705/storage-initializer/0.log"
Sep 17 18:57:08.655630 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.655601    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:57:08.835615 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.835578    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlfdq\" (UniqueName: \"kubernetes.io/projected/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-api-access-tlfdq\") pod \"b1159cdf-1d03-47ab-b93d-e5e252940705\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") "
Sep 17 18:57:08.835827 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.835623    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-cabundle-cert\") pod \"b1159cdf-1d03-47ab-b93d-e5e252940705\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") "
Sep 17 18:57:08.835827 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.835657    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1159cdf-1d03-47ab-b93d-e5e252940705-proxy-tls\") pod \"b1159cdf-1d03-47ab-b93d-e5e252940705\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") "
Sep 17 18:57:08.835827 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.835708    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1159cdf-1d03-47ab-b93d-e5e252940705-kserve-provision-location\") pod \"b1159cdf-1d03-47ab-b93d-e5e252940705\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") "
Sep 17 18:57:08.835827 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.835790    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-rbac-proxy-sar-config\") pod \"b1159cdf-1d03-47ab-b93d-e5e252940705\" (UID: \"b1159cdf-1d03-47ab-b93d-e5e252940705\") "
Sep 17 18:57:08.836118 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.836071    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1159cdf-1d03-47ab-b93d-e5e252940705" (UID: "b1159cdf-1d03-47ab-b93d-e5e252940705"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:08.836384 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.836357    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1159cdf-1d03-47ab-b93d-e5e252940705" (UID: "b1159cdf-1d03-47ab-b93d-e5e252940705"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:57:08.836519 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.836406    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1159cdf-1d03-47ab-b93d-e5e252940705-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1159cdf-1d03-47ab-b93d-e5e252940705" (UID: "b1159cdf-1d03-47ab-b93d-e5e252940705"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:57:08.837995 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.837975    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-api-access-tlfdq" (OuterVolumeSpecName: "kube-api-access-tlfdq") pod "b1159cdf-1d03-47ab-b93d-e5e252940705" (UID: "b1159cdf-1d03-47ab-b93d-e5e252940705"). InnerVolumeSpecName "kube-api-access-tlfdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:57:08.838087 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.838043    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1159cdf-1d03-47ab-b93d-e5e252940705-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1159cdf-1d03-47ab-b93d-e5e252940705" (UID: "b1159cdf-1d03-47ab-b93d-e5e252940705"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:57:08.937264 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.937224    2582 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-cabundle-cert\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:57:08.937264 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.937258    2582 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1159cdf-1d03-47ab-b93d-e5e252940705-proxy-tls\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:57:08.937264 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.937267    2582 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1159cdf-1d03-47ab-b93d-e5e252940705-kserve-provision-location\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:57:08.937264 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.937276    2582 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:57:08.937543 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:08.937285    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfdq\" (UniqueName: \"kubernetes.io/projected/b1159cdf-1d03-47ab-b93d-e5e252940705-kube-api-access-tlfdq\") on node \"ip-10-0-130-229.ec2.internal\" DevicePath \"\""
Sep 17 18:57:09.157718 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.157624    2582 generic.go:356] "Generic (PLEG): container finished" podID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerID="ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239" exitCode=0
Sep 17 18:57:09.157718 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.157671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" event={"ID":"b1159cdf-1d03-47ab-b93d-e5e252940705","Type":"ContainerDied","Data":"ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239"}
Sep 17 18:57:09.157718 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.157696    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"
Sep 17 18:57:09.158038 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.157725    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9" event={"ID":"b1159cdf-1d03-47ab-b93d-e5e252940705","Type":"ContainerDied","Data":"6a52983a8b6e9794c927780c576247a42407d603b252270d06ffe41f9bdd0a2d"}
Sep 17 18:57:09.158038 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.157754    2582 scope.go:117] "RemoveContainer" containerID="ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239"
Sep 17 18:57:09.166729 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.166710    2582 scope.go:117] "RemoveContainer" containerID="bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387"
Sep 17 18:57:09.174636 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.174618    2582 scope.go:117] "RemoveContainer" containerID="4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275"
Sep 17 18:57:09.180707 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.180682    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"]
Sep 17 18:57:09.183035 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.182985    2582 scope.go:117] "RemoveContainer" containerID="6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74"
Sep 17 18:57:09.185589 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.185566    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7dc457b548-npdh9"]
Sep 17 18:57:09.191542 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.191528    2582 scope.go:117] "RemoveContainer" containerID="ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239"
Sep 17 18:57:09.191819 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:57:09.191804    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239\": container with ID starting with ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239 not found: ID does not exist" containerID="ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239"
Sep 17 18:57:09.191865 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.191830    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239"} err="failed to get container status \"ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239\": rpc error: code = NotFound desc = could not find container \"ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239\": container with ID starting with ad9527bcf104628a1b052bb980fc4a6423260192423757443eadf182ef1c8239 not found: ID does not exist"
Sep 17 18:57:09.191865 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.191851    2582 scope.go:117] "RemoveContainer" containerID="bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387"
Sep 17 18:57:09.192137 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:57:09.192122    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387\": container with ID starting with bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387 not found: ID does not exist" containerID="bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387"
Sep 17 18:57:09.192195 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.192138    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387"} err="failed to get container status \"bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387\": rpc error: code = NotFound desc = could not find container \"bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387\": container with ID starting with bd2bf1abb9140e67791d6d03a3026fce8f0fc30784a5918414049bc8388bf387 not found: ID does not exist"
Sep 17 18:57:09.192195 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.192150    2582 scope.go:117] "RemoveContainer" containerID="4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275"
Sep 17 18:57:09.192402 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:57:09.192381    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275\": container with ID starting with 4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275 not found: ID does not exist" containerID="4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275"
Sep 17 18:57:09.192446 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.192409    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275"} err="failed to get container status \"4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275\": rpc error: code = NotFound desc = could not find container \"4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275\": container with ID starting with 4e08c7833ede19e3d6bc5139d42aae63579e515dfafc3de6f2099a0c0fb85275 not found: ID does not exist"
Sep 17 18:57:09.192446 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.192427    2582 scope.go:117] "RemoveContainer" containerID="6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74"
Sep 17 18:57:09.192683 ip-10-0-130-229 kubenswrapper[2582]: E0917 18:57:09.192665    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74\": container with ID starting with 6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74 not found: ID does not exist" containerID="6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74"
Sep 17 18:57:09.192724 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:09.192689    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74"} err="failed to get container status \"6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74\": rpc error: code = NotFound desc = could not find container \"6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74\": container with ID starting with 6e93156fd91e2605dfe28c839dc52b8cecb7666c72749e04f08aef7392875b74 not found: ID does not exist"
Sep 17 18:57:10.181828 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:10.181794    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" path="/var/lib/kubelet/pods/b1159cdf-1d03-47ab-b93d-e5e252940705/volumes"
Sep 17 18:57:14.818111 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:14.818075    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5v9qj_991b1e4c-1134-42a3-98d8-0c9f7a1385f2/global-pull-secret-syncer/0.log"
Sep 17 18:57:14.966587 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:14.966552    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rwlgb_676e15c7-92c5-4d69-b2d4-7cc98fe94e86/konnectivity-agent/0.log"
Sep 17 18:57:15.026665 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:15.026633    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-229.ec2.internal_41cc877ab68985cd82a0459700bd3eff/haproxy/0.log"
Sep 17 18:57:18.880058 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:18.880028    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6377d1a3-33b0-4c59-8a8e-1a7b24e639da/alertmanager/0.log"
Sep 17 18:57:18.913846 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:18.913818    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6377d1a3-33b0-4c59-8a8e-1a7b24e639da/config-reloader/0.log"
Sep 17 18:57:18.962610 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:18.962581    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6377d1a3-33b0-4c59-8a8e-1a7b24e639da/kube-rbac-proxy-web/0.log"
Sep 17 18:57:18.995977 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:18.995951    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6377d1a3-33b0-4c59-8a8e-1a7b24e639da/kube-rbac-proxy/0.log"
Sep 17 18:57:19.029375 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.029348    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6377d1a3-33b0-4c59-8a8e-1a7b24e639da/kube-rbac-proxy-metric/0.log"
Sep 17 18:57:19.057097 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.057072    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6377d1a3-33b0-4c59-8a8e-1a7b24e639da/prom-label-proxy/0.log"
Sep 17 18:57:19.082105 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.082080    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6377d1a3-33b0-4c59-8a8e-1a7b24e639da/init-config-reloader/0.log"
Sep 17 18:57:19.262262 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.262174    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8f958cdc9-clxc2_59c97eda-8507-4d0d-ac71-16d8dee61104/metrics-server/0.log"
Sep 17 18:57:19.436957 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.436932    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66p4g_a5081b60-bb6e-49d2-81c3-a1bba090bf97/node-exporter/0.log"
Sep 17 18:57:19.462157 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.462135    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66p4g_a5081b60-bb6e-49d2-81c3-a1bba090bf97/kube-rbac-proxy/0.log"
Sep 17 18:57:19.489384 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.489363    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66p4g_a5081b60-bb6e-49d2-81c3-a1bba090bf97/init-textfile/0.log"
Sep 17 18:57:19.614453 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.614426    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-k44fb_5e065573-72d8-4905-9f2a-b79416bd4f66/kube-rbac-proxy-main/0.log"
Sep 17 18:57:19.641825 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.641802    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-k44fb_5e065573-72d8-4905-9f2a-b79416bd4f66/kube-rbac-proxy-self/0.log"
Sep 17 18:57:19.670044 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.670018    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-k44fb_5e065573-72d8-4905-9f2a-b79416bd4f66/openshift-state-metrics/0.log"
Sep 17 18:57:19.927126 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.927029    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-hcr9d_56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7/prometheus-operator/0.log"
Sep 17 18:57:19.954242 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.954211    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-hcr9d_56d7d5a1-f6ea-4bc5-bf06-25eeb3ab67a7/kube-rbac-proxy/0.log"
Sep 17 18:57:19.982455 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:19.982424    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-rdxdl_df6f6550-661b-42f5-b580-3d43e73c40fc/prometheus-operator-admission-webhook/0.log"
Sep 17 18:57:20.016091 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:20.016064    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6fd699cdf4-sqn24_5ab102fd-f9dc-4955-9a6f-8b422689c64c/telemeter-client/0.log"
Sep 17 18:57:20.037725 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:20.037697    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6fd699cdf4-sqn24_5ab102fd-f9dc-4955-9a6f-8b422689c64c/reload/0.log"
Sep 17 18:57:20.060844 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:20.060816    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6fd699cdf4-sqn24_5ab102fd-f9dc-4955-9a6f-8b422689c64c/kube-rbac-proxy/0.log"
Sep 17 18:57:22.661287 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:22.661256    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-97876576-j7zsz_8a5d3b4d-6145-406a-b3e7-06697ecbb3bc/console/0.log"
Sep 17 18:57:22.696178 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:22.696147    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-rgx5v_7a187b07-eeb9-4516-9c07-3a3486fa2eb1/download-server/0.log"
Sep 17 18:57:23.238150 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238114    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"]
Sep 17 18:57:23.238473 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238461    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container"
Sep 17 18:57:23.238524 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238475    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container"
Sep 17 18:57:23.238524 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238483    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kube-rbac-proxy"
Sep 17 18:57:23.238524 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238488    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kube-rbac-proxy"
Sep 17 18:57:23.238524 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238498    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8e9238-21d8-43fe-96f1-d4ddb7f31064" containerName="kube-rbac-proxy"
Sep 17 18:57:23.238524 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238504    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8e9238-21d8-43fe-96f1-d4ddb7f31064" containerName="kube-rbac-proxy"
Sep 17 18:57:23.238524 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238518    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="storage-initializer"
Sep 17 18:57:23.238524 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238525    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="storage-initializer"
Sep 17 18:57:23.238731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238531    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8e9238-21d8-43fe-96f1-d4ddb7f31064" containerName="kserve-container"
Sep 17 18:57:23.238731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238536    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8e9238-21d8-43fe-96f1-d4ddb7f31064" containerName="kserve-container"
Sep 17 18:57:23.238731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238546    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent"
Sep 17 18:57:23.238731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238551    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent"
Sep 17 18:57:23.238731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238605    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kube-rbac-proxy"
Sep 17 18:57:23.238731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238612    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="agent"
Sep 17 18:57:23.238731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238620    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8e9238-21d8-43fe-96f1-d4ddb7f31064" containerName="kserve-container"
Sep 17 18:57:23.238731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238626    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8e9238-21d8-43fe-96f1-d4ddb7f31064" containerName="kube-rbac-proxy"
Sep 17 18:57:23.238731 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.238632    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1159cdf-1d03-47ab-b93d-e5e252940705" containerName="kserve-container"
Sep 17 18:57:23.241677 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.241658    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.244386 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.244370    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b6x4n\"/\"kube-root-ca.crt\""
Sep 17 18:57:23.244681 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.244662    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b6x4n\"/\"openshift-service-ca.crt\""
Sep 17 18:57:23.245143 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.245127    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b6x4n\"/\"default-dockercfg-tjq4r\""
Sep 17 18:57:23.251108 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.251080    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"]
Sep 17 18:57:23.354810 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.354773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-podres\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.354810 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.354814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-lib-modules\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.355095 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.354923    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-sys\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.355095 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.354970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-proc\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.355095 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.355039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxst\" (UniqueName: \"kubernetes.io/projected/8199b4e7-d914-4130-8e30-a83e38c1de64-kube-api-access-twxst\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.456437 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.456402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxst\" (UniqueName: \"kubernetes.io/projected/8199b4e7-d914-4130-8e30-a83e38c1de64-kube-api-access-twxst\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.456611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.456457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-podres\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.456611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.456494    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-lib-modules\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.456611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.456561    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-sys\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.456611 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.456591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-proc\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.456803 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.456640    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-podres\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.456803 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.456674    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-proc\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.456803 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.456709    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-sys\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.456803 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.456718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8199b4e7-d914-4130-8e30-a83e38c1de64-lib-modules\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.465146 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.465105    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxst\" (UniqueName: \"kubernetes.io/projected/8199b4e7-d914-4130-8e30-a83e38c1de64-kube-api-access-twxst\") pod \"perf-node-gather-daemonset-dgcml\" (UID: \"8199b4e7-d914-4130-8e30-a83e38c1de64\") " pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.553180 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.553152    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:23.824720 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:23.824689    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"]
Sep 17 18:57:23.827321 ip-10-0-130-229 kubenswrapper[2582]: W0917 18:57:23.827288    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8199b4e7_d914_4130_8e30_a83e38c1de64.slice/crio-b6e8eaed4c7dba5e9eec61fe151bb68d16e375951e053c94da66a0ad31a632c5 WatchSource:0}: Error finding container b6e8eaed4c7dba5e9eec61fe151bb68d16e375951e053c94da66a0ad31a632c5: Status 404 returned error can't find the container with id b6e8eaed4c7dba5e9eec61fe151bb68d16e375951e053c94da66a0ad31a632c5
Sep 17 18:57:24.228984 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:24.228862    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml" event={"ID":"8199b4e7-d914-4130-8e30-a83e38c1de64","Type":"ContainerStarted","Data":"56f9fd39b4ede1811a337dec23274e67422b211543e6eed89539df0ac4d7cb3b"}
Sep 17 18:57:24.228984 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:24.228960    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml" event={"ID":"8199b4e7-d914-4130-8e30-a83e38c1de64","Type":"ContainerStarted","Data":"b6e8eaed4c7dba5e9eec61fe151bb68d16e375951e053c94da66a0ad31a632c5"}
Sep 17 18:57:24.229166 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:24.229002    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:24.246006 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:24.245948    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml" podStartSLOduration=1.245931725 podStartE2EDuration="1.245931725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:57:24.243944145 +0000 UTC m=+1678.167726807" watchObservedRunningTime="2026-09-17 18:57:24.245931725 +0000 UTC m=+1678.169714385"
Sep 17 18:57:24.283666 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:24.283642    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gs8xz_bb7e34a0-916a-48d9-ab80-b75deacc2223/dns/0.log"
Sep 17 18:57:24.307467 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:24.307413    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gs8xz_bb7e34a0-916a-48d9-ab80-b75deacc2223/kube-rbac-proxy/0.log"
Sep 17 18:57:24.335493 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:24.335463    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4brlg_a76a82b7-6e16-4354-92aa-da9e6b01b704/dns-node-resolver/0.log"
Sep 17 18:57:24.859790 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:24.859756    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f95f5465-8hsbr_9567c646-0644-44f7-928f-d89851032606/registry/0.log"
Sep 17 18:57:24.880079 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:24.880055    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p75rj_3ef8e676-9989-483a-be1f-751f64eddfe7/node-ca/0.log"
Sep 17 18:57:26.080539 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:26.080508    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bmmdz_0050eb62-8496-4d2a-80ba-e4204df7e439/serve-healthcheck-canary/0.log"
Sep 17 18:57:26.613652 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:26.613622    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-khc8v_bf5e7039-0562-4578-a7bf-5b061c217ce1/kube-rbac-proxy/0.log"
Sep 17 18:57:26.635353 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:26.635327    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-khc8v_bf5e7039-0562-4578-a7bf-5b061c217ce1/exporter/0.log"
Sep 17 18:57:26.657106 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:26.657078    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-khc8v_bf5e7039-0562-4578-a7bf-5b061c217ce1/extractor/0.log"
Sep 17 18:57:28.964983 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:28.964929    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-69d59d7cdd-cbhgq_75fdea0a-c437-4815-8060-0251f13a2bc8/manager/0.log"
Sep 17 18:57:29.132576 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:29.132537    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-2vfps_0cca4acc-60f9-4160-a8e3-46b862f27304/manager/0.log"
Sep 17 18:57:29.233938 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:29.233829    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-t7jdl_9b7741da-1844-4b58-b52b-109a9e69560a/seaweedfs/0.log"
Sep 17 18:57:30.244393 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:30.244361    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b6x4n/perf-node-gather-daemonset-dgcml"
Sep 17 18:57:33.559281 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:33.559253    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-k589s_ea8a74cf-b930-4861-94f1-a80b8c54758b/migrator/0.log"
Sep 17 18:57:33.582458 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:33.582424    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-k589s_ea8a74cf-b930-4861-94f1-a80b8c54758b/graceful-termination/0.log"
Sep 17 18:57:35.001787 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:35.001714    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5259j_d6c904bd-89ec-40d5-88f4-ba4ce4d194fa/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:57:35.033369 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:35.033336    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5259j_d6c904bd-89ec-40d5-88f4-ba4ce4d194fa/egress-router-binary-copy/0.log"
Sep 17 18:57:35.061982 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:35.061958    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5259j_d6c904bd-89ec-40d5-88f4-ba4ce4d194fa/cni-plugins/0.log"
Sep 17 18:57:35.088559 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:35.088535    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5259j_d6c904bd-89ec-40d5-88f4-ba4ce4d194fa/bond-cni-plugin/0.log"
Sep 17 18:57:35.116909 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:35.116871    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5259j_d6c904bd-89ec-40d5-88f4-ba4ce4d194fa/routeoverride-cni/0.log"
Sep 17 18:57:35.143654 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:35.143627    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5259j_d6c904bd-89ec-40d5-88f4-ba4ce4d194fa/whereabouts-cni-bincopy/0.log"
Sep 17 18:57:35.175599 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:35.175574    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5259j_d6c904bd-89ec-40d5-88f4-ba4ce4d194fa/whereabouts-cni/0.log"
Sep 17 18:57:35.665429 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:35.665395    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-smrws_a11f2d11-4fd9-4ed2-bb2b-cc18a208a38a/kube-multus/0.log"
Sep 17 18:57:35.747613 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:35.747582    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rsp6q_b3ef2bd6-26b1-447e-9b89-aae0cf51d91c/network-metrics-daemon/0.log"
Sep 17 18:57:35.769719 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:35.769694    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rsp6q_b3ef2bd6-26b1-447e-9b89-aae0cf51d91c/kube-rbac-proxy/0.log"
Sep 17 18:57:37.234146 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:37.234113    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p9vs8_0755d8ba-cf82-4e44-9f74-1c53bebedf8c/ovn-controller/0.log"
Sep 17 18:57:37.256814 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:37.256790    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p9vs8_0755d8ba-cf82-4e44-9f74-1c53bebedf8c/ovn-acl-logging/0.log"
Sep 17 18:57:37.283114 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:37.283088    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p9vs8_0755d8ba-cf82-4e44-9f74-1c53bebedf8c/kube-rbac-proxy-node/0.log"
Sep 17 18:57:37.305632 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:37.305605    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p9vs8_0755d8ba-cf82-4e44-9f74-1c53bebedf8c/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:57:37.330149 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:37.330123    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p9vs8_0755d8ba-cf82-4e44-9f74-1c53bebedf8c/northd/0.log"
Sep 17 18:57:37.353654 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:37.353627    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p9vs8_0755d8ba-cf82-4e44-9f74-1c53bebedf8c/nbdb/0.log"
Sep 17 18:57:37.379367 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:37.379346    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p9vs8_0755d8ba-cf82-4e44-9f74-1c53bebedf8c/sbdb/0.log"
Sep 17 18:57:37.421283 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:37.421252    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p9vs8_0755d8ba-cf82-4e44-9f74-1c53bebedf8c/ovnkube-controller/0.log"
Sep 17 18:57:38.435404 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:38.435313    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dt469_24e1408d-b93a-449f-bd8e-6dfb76828d48/network-check-target-container/0.log"
Sep 17 18:57:39.397306 ip-10-0-130-229 kubenswrapper[2582]: I0917 18:57:39.397270    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8tph9_8528fdf6-c48e-4f4d-bdc4-a685de05b47d/iptables-alerter/0.log"