Sep 17 21:58:19.448414 ip-10-0-130-119 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 21:58:22.937480 ip-10-0-130-119 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:22.937480 ip-10-0-130-119 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 21:58:22.937480 ip-10-0-130-119 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:22.937480 ip-10-0-130-119 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 21:58:22.937480 ip-10-0-130-119 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 21:58:22.938279 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.937547    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 21:58:22.941366 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941350    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:22.941366 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941366    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941371    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941375    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941378    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941381    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941384    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941388    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941391    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941393    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941396    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941399    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941401    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941404    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941407    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941409    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941412    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941415    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941417    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941420    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941422    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:22.941428 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941425    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941433    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941436    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941439    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941441    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941444    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941447    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941449    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941452    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941454    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941457    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941459    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941464    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941467    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941470    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941473    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941476    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941479    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941481    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941484    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:22.941935 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941488    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941490    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941492    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941495    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941497    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941500    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941503    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941505    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941508    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941510    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941512    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941515    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941518    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941522    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941526    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941529    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941532    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941535    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941537    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:22.942444 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941540    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941543    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941545    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941548    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941550    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941553    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941555    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941557    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941560    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941562    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941565    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941568    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941571    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941573    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941576    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941578    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941581    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941583    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941586    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941588    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:22.942911 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941590    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941593    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941595    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941598    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941601    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941604    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941606    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941609    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941612    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941615    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941617    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941620    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941623    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941625    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941628    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941630    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941633    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941636    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.941640    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942348    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:22.943412 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942354    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942357    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942360    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942363    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942366    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942369    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942371    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942375    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942378    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942380    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942383    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942386    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942388    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942391    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942394    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942397    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942400    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942402    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942405    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942409    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:22.943895 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942413    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942415    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942418    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942421    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942424    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942427    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942429    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942432    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942435    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942437    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942440    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942443    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942446    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942449    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942451    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942454    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942457    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942462    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942466    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942469    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:22.944398 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942472    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942475    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942478    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942480    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942484    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942488    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942491    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942494    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942496    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942499    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942501    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942504    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942507    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942510    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942512    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942515    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942517    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942520    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942523    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942525    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:22.944889 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942528    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942530    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942533    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942536    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942538    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942541    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942543    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942546    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942549    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942551    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942554    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942558    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942562    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942565    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942568    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942571    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942574    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942577    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942580    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942583    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:22.945396 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942586    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942588    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942591    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942595    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942598    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942601    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942603    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942606    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942609    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942611    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942614    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942617    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942619    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942622    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942625    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942628    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942631    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.942633    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943007    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943012    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:22.945931 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943015    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943017    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943021    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943023    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943026    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943029    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943033    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943038    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943041    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943044    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943047    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943050    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943053    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943056    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943059    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943062    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943065    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943067    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943070    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:22.946449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943073    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943075    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943077    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943080    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943083    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943085    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943088    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943090    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943093    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943095    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943098    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943101    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943103    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943106    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943108    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943111    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943114    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943116    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943119    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943121    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:22.946948 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943124    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943126    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943129    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943131    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943151    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943154    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943156    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943159    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943165    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943169    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943172    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943174    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943177    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943179    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943182    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943185    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943187    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943189    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943192    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943195    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:22.947442 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943197    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943200    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943202    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943205    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943207    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943210    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943212    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943215    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943217    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943221    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943224    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943227    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943229    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943232    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943234    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943237    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943239    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943242    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943244    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943246    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:22.947939 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943251    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943253    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943256    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943258    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943261    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943264    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943267    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943269    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943272    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943275    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943277    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943280    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943282    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943285    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943287    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943290    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943292    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.943295    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943369    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943402    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943408    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 21:58:22.948486 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943412    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943418    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943421    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943425    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943430    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943433    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943436    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943440    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943443    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943446    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943450    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943453    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943457    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943459    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943462    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943466    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943469    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943472    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943475    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943479    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943482    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943485    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943489    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943492    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 21:58:22.948989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943495    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943498    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943501    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943504    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943507    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943511    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943514    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943517    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943520    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943523    2583 flags.go:64] FLAG: --enable-server="true"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943526    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943530    2583 flags.go:64] FLAG: --event-burst="100"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943534    2583 flags.go:64] FLAG: --event-qps="50"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943537    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943540    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943543    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943547    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943549    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943552    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943556    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943559    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943562    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943566    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943569    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943572    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943575    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 21:58:22.949580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943577    2583 flags.go:64] FLAG: --feature-gates=""
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943581    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943584    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943587    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943591    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943594    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943597    2583 flags.go:64] FLAG: --help="false"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943600    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-130-119.ec2.internal"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943603    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943606    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943609    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943612    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943616    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943619    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943622    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943625    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943627    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943631    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943634    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943637    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943640    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943643    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943646    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943648    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 21:58:22.950276 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943651    2583 flags.go:64] FLAG: --lock-file=""
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943656    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943659    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943663    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943668    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943671    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943674    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943676    2583 flags.go:64] FLAG: --logging-format="text"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943679    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943682    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943686    2583 flags.go:64] FLAG: --manifest-url=""
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943688    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943693    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943695    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943699    2583 flags.go:64] FLAG: --max-pods="110"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943702    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943705    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943709    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943712    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943715    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943718    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943721    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943727    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943730    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943733    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 21:58:22.950888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943736    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943740    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943744    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943747    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943751    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943754    2583 flags.go:64] FLAG: --port="10250"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943757    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943760    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03142099fefe25491"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943765    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943768    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943771    2583 flags.go:64] FLAG: --register-node="true"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943774    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943778    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943781    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943783    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943786    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943790    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943793    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943796    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943799    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943801    2583 flags.go:64] FLAG: --runonce="false"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943804    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943807    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943810    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943813    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 21:58:22.951526 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943816    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943819    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943822    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943825    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943828    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943831    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943833    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943836    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943839    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943842    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943845    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943850    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943853    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943856    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943859    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943863    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943868    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943871    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943874    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943877    2583 flags.go:64] FLAG: --v="2"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943881    2583 flags.go:64] FLAG: --version="false"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943884    2583 flags.go:64] FLAG: --vmodule=""
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943888    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.943891    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944036    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:22.952176 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944040    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944043    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944046    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944049    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944051    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944054    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944057    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944060    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944063    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944065    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944068    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944072    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944074    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944077    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944079    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944082    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944085    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944088    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944091    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944093    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:22.952777 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944096    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944099    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944103    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944107    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944111    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944114    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944118    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944121    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944124    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944127    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944130    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944146    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944149    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944152    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944155    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944157    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944160    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944163    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944166    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:22.953279 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944168    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944171    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944174    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944177    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944180    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944183    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944185    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944188    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944191    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944193    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944196    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944199    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944202    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944205    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944207    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944211    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944215    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944218    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944220    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944223    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:22.953743 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944226    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944228    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944231    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944234    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944236    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944239    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944241    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944245    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944249    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944252    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944255    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944257    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944260    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944263    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944266    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944270    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944273    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944276    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944279    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:22.954261 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944281    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944284    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944287    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944290    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944292    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944295    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944298    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944300    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944305    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944309    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944312    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944314    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944317    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944320    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944322    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944325    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944327    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944330    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944333    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:22.954735 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.944335    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.944852    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.951478    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.951493    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.951510    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.951519    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951575    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951580    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951583    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951587    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951590    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951593    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951596    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951599    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951602    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:22.955256 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951604    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951607    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951610    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951613    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951616    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951618    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951621    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951624    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951626    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951629    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951633    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951637    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951640    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951644    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951646    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951649    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951653    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951655    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951658    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:22.955631 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951662    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951667    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951670    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951673    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951676    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951679    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951682    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951697    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951700    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951704    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951707    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951709    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951712    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951715    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951718    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951721    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951723    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951726    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951729    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951731    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951734    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:22.956105 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951737    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951739    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951742    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951745    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951747    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951750    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951753    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951755    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951758    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951760    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951763    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951766    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951769    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951772    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951774    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951777    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951780    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951782    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951785    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:22.956623 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951788    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951790    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951793    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951795    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951798    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951802    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951805    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951808    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951811    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951814    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951817    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951819    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951822    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951825    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951828    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951831    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951834    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951837    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951840    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951843    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:22.957089 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951845    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951848    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951850    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951853    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951856    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951859    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951861    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951864    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951867    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951870    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951872    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.951878    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951987    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951992    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951995    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.951998    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 21:58:22.957652 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952001    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952004    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952006    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952009    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952012    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952014    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952017    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952020    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952022    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952025    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952027    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952030    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952032    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952036    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952039    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952041    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952044    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952046    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952049    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952052    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 21:58:22.958036 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952054    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952057    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952060    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952062    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952066    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952068    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952071    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952074    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952076    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952079    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952083    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952087    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952089    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952092    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952095    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952097    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952100    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952103    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952105    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952108    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 21:58:22.958524 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952110    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952113    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952115    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952118    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952120    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952124    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952126    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952129    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952131    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952149    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952152    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952155    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952157    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952160    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952163    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952166    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952169    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952172    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952175    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952179    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 21:58:22.959024 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952183    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952186    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952189    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952192    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952194    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952197    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952199    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952202    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952204    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952207    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952210    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952212    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952215    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952217    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952220    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952222    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952225    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952228    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952231    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952233    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 21:58:22.959618 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952236    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952238    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952241    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952244    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952247    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952250    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952253    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952256    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952258    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952261    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952264    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952266    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952269    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952271    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 21:58:22.960295 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:22.952274    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 21:58:22.960836 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.952279    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 21:58:22.960836 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.952407    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 21:58:22.960836 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.954376    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 21:58:22.960836 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.955289    2583 server.go:1019] "Starting client certificate rotation"
Sep 17 21:58:22.960836 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.955386    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:22.960836 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.955915    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 21:58:22.960836 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.957942    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:22.960836 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.959312    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 21:58:22.979004 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.978979    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 17 21:58:22.980585 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.980572    2583 log.go:25] "Validated CRI v1 image API"
Sep 17 21:58:22.981363 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.981348    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 21:58:22.985511 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.985491    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ad17e7df-a5a6-4823-bb05-d86fc15719c6:/dev/nvme0n1p4 b95a9670-6446-4ea7-a0f3-a2d810a3e5d0:/dev/nvme0n1p3]
Sep 17 21:58:22.985583 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.985508    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 21:58:22.990519 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.990404    2583 manager.go:216] Machine: {Timestamp:2026-09-17 21:58:22.989925646 +0000 UTC m=+2.073513607 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3128261 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fd0e656b73c4a2a8036d4eda48e00 SystemUUID:ec2fd0e6-56b7-3c4a-2a80-36d4eda48e00 BootID:5cb4862a-1347-4a71-a6bb-e4e4ee888e59 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:e0:90:37 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:e0:90:37 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:d4:7d:72:37:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 21:58:22.990519 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.990511    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 21:58:22.990635 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.990606    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 21:58:22.992080 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.992039    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 21:58:22.992226 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.992081    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-119.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 21:58:22.993246 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.993236    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 21:58:22.993279 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.993248    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 21:58:22.993279 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.993255    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:22.993279 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.993266    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 21:58:22.993381 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.993372    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 21:58:22.995988 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.995976    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:22.996103 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.996093    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 21:58:22.996158 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.996129    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 21:58:22.996158 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.996155    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 21:58:22.996216 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.996177    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 21:58:22.996216 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.996190    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 17 21:58:22.996216 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.996202    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 21:58:22.996599 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.996581    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:22.996662 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.996608    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 21:58:22.999115 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.999066    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 21:58:22.999193 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:22.999099    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:23.001284 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.001268    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 21:58:23.001320 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.001302    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 21:58:23.002301 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002287    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 21:58:23.002352 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002305    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 21:58:23.002352 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002312    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 21:58:23.002352 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002317    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 21:58:23.002352 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002325    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 21:58:23.002352 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002331    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 21:58:23.002352 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002345    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 21:58:23.002352 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002349    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 21:58:23.002534 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002360    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 21:58:23.002534 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002365    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 21:58:23.002534 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002373    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 21:58:23.002534 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002378    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 21:58:23.002534 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002399    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 21:58:23.002534 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.002404    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 21:58:23.006488 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.006473    2583 server.go:1268] "Started kubelet"
Sep 17 21:58:23.007204 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.007154    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 21:58:23.007312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.007234    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 21:58:23.007412 ip-10-0-130-119 systemd[1]: Started Kubernetes Kubelet.
Sep 17 21:58:23.036523 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.007005    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 21:58:23.036919 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.036896    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-119.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 21:58:23.036983 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.036490    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-119.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 21:58:23.037054 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.037035    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sd8sh"
Sep 17 21:58:23.037099 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.037057    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sd8sh"
Sep 17 21:58:23.037443 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.036277    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 21:58:23.037570 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.037550    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 21:58:23.040314 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.040290    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 21:58:23.042660 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.041733    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-119.ec2.internal.18d63a928b7f999c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-119.ec2.internal,UID:ip-10-0-130-119.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-119.ec2.internal,},FirstTimestamp:2026-09-17 21:58:23.006382492 +0000 UTC m=+2.089970433,LastTimestamp:2026-09-17 21:58:23.006382492 +0000 UTC m=+2.089970433,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-119.ec2.internal,}"
Sep 17 21:58:23.044617 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.044599    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 21:58:23.044815 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.044799    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 21:58:23.044900 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.044819    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:23.044962 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.044946    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:23.045496 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.045477    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 21:58:23.045578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.045500    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 21:58:23.045578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.045477    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 21:58:23.045675 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.045627    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 21:58:23.045675 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.045633    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 21:58:23.046708 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.046691    2583 factory.go:153] Registering CRI-O factory
Sep 17 21:58:23.046708 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.046710    2583 factory.go:223] Registration of the crio container factory successfully
Sep 17 21:58:23.046841 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.046801    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 21:58:23.046841 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.046811    2583 factory.go:55] Registering systemd factory
Sep 17 21:58:23.046841 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.046819    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 17 21:58:23.046841 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.046838    2583 factory.go:103] Registering Raw factory
Sep 17 21:58:23.047016 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.046851    2583 manager.go:1196] Started watching for new ooms in manager
Sep 17 21:58:23.047315 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.047300    2583 manager.go:349] Starting recovery of all containers
Sep 17 21:58:23.054702 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.054635    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:23.057537 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.057519    2583 manager.go:354] Recovery completed
Sep 17 21:58:23.057609 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.057564    2583 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-119.ec2.internal\" not found" node="ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.061907 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.061894    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:23.063748 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.063734    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:23.063821 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.063761    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:23.063821 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.063773    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:23.064322 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.064308    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 21:58:23.064322 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.064321    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 21:58:23.064414 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.064340    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 21:58:23.064414 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.064310    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:23.065128 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.065107    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:23.065237 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.065181    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:23.065237 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.065194    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:23.065595 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.065580    2583 policy_none.go:49] "None policy: Start"
Sep 17 21:58:23.065595 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.065595    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 21:58:23.065709 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.065607    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 21:58:23.066784 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.066768    2583 policy_none.go:47] "Start"
Sep 17 21:58:23.132696 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.132676    2583 manager.go:337] "Starting Device Plugin manager"
Sep 17 21:58:23.132816 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.132778    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 21:58:23.132816 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.132789    2583 server.go:87] "Starting device plugin registration server"
Sep 17 21:58:23.133048 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.133036    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 21:58:23.133098 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.133058    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 21:58:23.133191 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.133175    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 21:58:23.133293 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.133274    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 21:58:23.133293 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.133292    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 21:58:23.134276 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.134260    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 21:58:23.134343 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.134305    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:23.178488 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.178456    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 21:58:23.179698 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.179678    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 21:58:23.179805 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.179724    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 21:58:23.179805 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.179744    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 21:58:23.179805 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.179787    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 21:58:23.182217 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.182193    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:23.233253 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.233185    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:23.234215 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.234200    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:23.234304 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.234232    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:23.234304 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.234250    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:23.234304 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.234282    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.242565 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.242550    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.242643 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.242571    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-119.ec2.internal\": node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:23.255747 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.255728    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:23.280211 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.280190    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"]
Sep 17 21:58:23.280279 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.280274    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:23.281306 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.281291    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:23.281378 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.281319    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:23.281378 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.281329    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:23.282530 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.282517    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:23.282667 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.282652    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.282725 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.282686    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:23.283216 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.283197    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:23.283318 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.283202    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:23.283318 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.283223    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:23.283318 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.283244    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:23.283318 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.283255    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:23.283318 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.283246    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:23.284263 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.284246    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.284352 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.284273    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 21:58:23.284982 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.284969    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 21:58:23.285056 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.284997    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 21:58:23.285056 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.285011    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 17 21:58:23.309787 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.309763    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-119.ec2.internal\" not found" node="ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.314363 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.314347    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-119.ec2.internal\" not found" node="ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.347200 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.347177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a6232eedc32fbab34ed57bf917cc434-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"6a6232eedc32fbab34ed57bf917cc434\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.347283 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.347213    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a6232eedc32fbab34ed57bf917cc434-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"6a6232eedc32fbab34ed57bf917cc434\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.347283 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.347239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca36c8d3176c4f0a591a991efcdaef22-config\") pod \"kube-apiserver-proxy-ip-10-0-130-119.ec2.internal\" (UID: \"ca36c8d3176c4f0a591a991efcdaef22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.356222 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.356203    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:23.448257 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.448226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a6232eedc32fbab34ed57bf917cc434-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"6a6232eedc32fbab34ed57bf917cc434\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.448257 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.448258    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a6232eedc32fbab34ed57bf917cc434-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"6a6232eedc32fbab34ed57bf917cc434\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.448422 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.448278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca36c8d3176c4f0a591a991efcdaef22-config\") pod \"kube-apiserver-proxy-ip-10-0-130-119.ec2.internal\" (UID: \"ca36c8d3176c4f0a591a991efcdaef22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.448422 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.448334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a6232eedc32fbab34ed57bf917cc434-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"6a6232eedc32fbab34ed57bf917cc434\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.448422 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.448364    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca36c8d3176c4f0a591a991efcdaef22-config\") pod \"kube-apiserver-proxy-ip-10-0-130-119.ec2.internal\" (UID: \"ca36c8d3176c4f0a591a991efcdaef22\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.448422 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.448334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a6232eedc32fbab34ed57bf917cc434-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"6a6232eedc32fbab34ed57bf917cc434\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.456281 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.456255    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:23.557067 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.556993    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:23.613211 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.613178    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.617741 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.617624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:23.657928 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.657905    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:23.758399 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.758358    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:23.858963 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.858899    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:23.955432 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.955403    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 21:58:23.956028 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.955543    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:23.956028 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.955549    2583 request.go:1196] "Unexpected error when reading response body" err="read tcp 10.0.130.119:43378->75.101.149.133:6443: use of closed network connection"
Sep 17 21:58:23.956028 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:23.955561    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 21:58:23.956028 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.955614    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: unexpected error when reading response body. Please retry. Original error: read tcp 10.0.130.119:43378->75.101.149.133:6443: use of closed network connection" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 21:58:23.959661 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:23.959643    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:24.041006 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:24.040967    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 21:53:23 +0000 UTC" deadline="2028-06-26 11:37:57.870246486 +0000 UTC"
Sep 17 21:58:24.041006 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:24.040998    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15541h39m33.829252142s"
Sep 17 21:58:24.059749 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:24.059718    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:24.160342 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:24.160317    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:24.261334 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:24.261299    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:24.306881 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:24.306841    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca36c8d3176c4f0a591a991efcdaef22.slice/crio-8ffee43ca49aca01361bba4f4253494c29714b85de0af41b1421fe56fdacc9e7 WatchSource:0}: Error finding container 8ffee43ca49aca01361bba4f4253494c29714b85de0af41b1421fe56fdacc9e7: Status 404 returned error can't find the container with id 8ffee43ca49aca01361bba4f4253494c29714b85de0af41b1421fe56fdacc9e7
Sep 17 21:58:24.307186 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:24.307172    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6232eedc32fbab34ed57bf917cc434.slice/crio-fb52d7b3bf9d588023afa6e84b82690e86631899c28a315aa0b73b0741aa37b5 WatchSource:0}: Error finding container fb52d7b3bf9d588023afa6e84b82690e86631899c28a315aa0b73b0741aa37b5: Status 404 returned error can't find the container with id fb52d7b3bf9d588023afa6e84b82690e86631899c28a315aa0b73b0741aa37b5
Sep 17 21:58:24.311832 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:24.311815    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:58:24.361750 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:24.361727    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:24.462420 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:24.462332    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:24.562961 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:24.562932    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 17 21:58:24.608209 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:24.608183    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:24.645202 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:24.645175    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:24.654702 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:24.654679    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:24.656398 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:24.656384    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 17 21:58:24.665199 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:24.665182    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 21:58:24.923083 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:24.922993    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:24.997731 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:24.997693    2583 apiserver.go:52] "Watching apiserver"
Sep 17 21:58:25.005287 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.005264    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 21:58:25.006911 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.006876    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rffj6","kube-system/konnectivity-agent-lf995","openshift-cluster-node-tuning-operator/tuned-wcphq","openshift-dns/node-resolver-f4lz4","openshift-image-registry/node-ca-qfs4z","openshift-multus/multus-2n6n9","openshift-multus/multus-additional-cni-plugins-wlb9z","openshift-multus/network-metrics-daemon-n64gh","kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal","openshift-network-diagnostics/network-check-target-q8xvj","openshift-network-operator/iptables-alerter-qqc5b","openshift-ovn-kubernetes/ovnkube-node-vpmct"]
Sep 17 21:58:25.008908 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.008887    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.010317 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.010301    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qqc5b"
Sep 17 21:58:25.011613 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.011595    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.012024 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.012010    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 21:58:25.012094 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.012039    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qvz9s\""
Sep 17 21:58:25.012162 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.012124    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 21:58:25.012256 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.012236    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 21:58:25.012404 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.012382    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 21:58:25.012471 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.012414    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 21:58:25.012711 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.012694    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:25.012988 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.012803    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 21:58:25.012988 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.012910    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cjw6z\""
Sep 17 21:58:25.012988 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.012907    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:25.013541 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.013526    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:58:25.014424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.014148    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f4lz4"
Sep 17 21:58:25.014424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.014157    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 21:58:25.015319 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.015298    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qfs4z"
Sep 17 21:58:25.016049 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.015912    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bnq92\""
Sep 17 21:58:25.017043 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.017006    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.018052 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.018034    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 21:58:25.019059 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.018344    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.019059 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.018956    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 21:58:25.019217 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.019196    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 21:58:25.019217 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.019204    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 21:58:25.019509 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.019318    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 21:58:25.019509 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.019360    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j7b9c\""
Sep 17 21:58:25.019509 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.019419    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m9t96\""
Sep 17 21:58:25.019509 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.019515    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kqwlt\""
Sep 17 21:58:25.019835 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.019609    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 21:58:25.020772 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.020390    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:25.020772 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.020467    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:25.021662 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.021020    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 21:58:25.021662 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.021230    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 21:58:25.021662 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.021245    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 21:58:25.021662 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.021249    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 21:58:25.021662 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.021404    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 21:58:25.021662 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.021465    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dgcjl\""
Sep 17 21:58:25.021662 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.021515    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 21:58:25.022337 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.022317    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:25.023961 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.023942    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.024619 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.024600    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 21:58:25.024956 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.024933    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rnrjz\""
Sep 17 21:58:25.025068 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.025053    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 21:58:25.025667 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.025648    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:25.025744 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.025718    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:25.026306 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.026288    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 21:58:25.026420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.026397    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 21:58:25.027164 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.026707    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 21:58:25.027164 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.026713    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wg9ms\""
Sep 17 21:58:25.027164 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.026984    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:25.027164 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.027039    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:25.045742 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.045720    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 21:58:25.046775 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.046755    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 21:58:25.057575 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bhhh\" (UniqueName: \"kubernetes.io/projected/2acdf951-0d97-4bb7-bbce-cc70bddb8e47-kube-api-access-4bhhh\") pod \"node-ca-qfs4z\" (UID: \"2acdf951-0d97-4bb7-bbce-cc70bddb8e47\") " pod="openshift-image-registry/node-ca-qfs4z"
Sep 17 21:58:25.057726 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfrqx\" (UniqueName: \"kubernetes.io/projected/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-kube-api-access-jfrqx\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.057816 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057801    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6df8b67b-5e30-4cbb-8495-6f78f2524e13-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.057894 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057815    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 21:58:25.057894 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-systemd\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.057894 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-device-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.057894 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-conf-dir\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.058082 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-tuned\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.058082 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057923    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-run-k8s-cni-cncf-io\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.058082 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-run-netns\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.058082 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057978    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-kubelet\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.058082 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.057999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-etc-openvswitch\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.058082 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058022    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-log-socket\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.058082 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6df8b67b-5e30-4cbb-8495-6f78f2524e13-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.058082 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-var-lib-kubelet\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.058472 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a921b3aa-f930-4ac3-9cb4-9af843a381df-tmp\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.058472 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74xkq\" (UniqueName: \"kubernetes.io/projected/a921b3aa-f930-4ac3-9cb4-9af843a381df-kube-api-access-74xkq\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.058472 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:25.058472 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.058472 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058299    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-cni-netd\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.058472 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-784fz\" (UniqueName: \"kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz\") pod \"network-check-target-q8xvj\" (UID: \"032d58e2-2e31-40c8-87cc-a4d5939db410\") " pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:25.058472 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6df8b67b-5e30-4cbb-8495-6f78f2524e13-cni-binary-copy\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.058472 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f93f48f-8169-4a8b-885c-b591f433f601-kubelet-config\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:25.058472 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058435    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-registration-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.058472 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-etc-kubernetes\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-ovnkube-script-lib\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1e37671-3221-44a1-b4c0-20f45cec8dcf-konnectivity-ca\") pod \"konnectivity-agent-lf995\" (UID: \"c1e37671-3221-44a1-b4c0-20f45cec8dcf\") " pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-run-openvswitch\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdz7g\" (UniqueName: \"kubernetes.io/projected/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-kube-api-access-sdz7g\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058647    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kmwt\" (UniqueName: \"kubernetes.io/projected/6df8b67b-5e30-4cbb-8495-6f78f2524e13-kube-api-access-6kmwt\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-modprobe-d\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-host\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058730    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-socket-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrvnx\" (UniqueName: \"kubernetes.io/projected/425aa0ec-46a3-4712-9a04-91b742dde2b6-kube-api-access-lrvnx\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058776    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-hostroot\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6df8b67b-5e30-4cbb-8495-6f78f2524e13-cnibin\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-sysconfig\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxtjq\" (UniqueName: \"kubernetes.io/projected/e9d7824d-7ae5-434d-8ccc-decace1ecd67-kube-api-access-wxtjq\") pod \"node-resolver-f4lz4\" (UID: \"e9d7824d-7ae5-434d-8ccc-decace1ecd67\") " pod="openshift-dns/node-resolver-f4lz4"
Sep 17 21:58:25.058940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-cnibin\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.058972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-daemon-config\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059016    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-node-log\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6df8b67b-5e30-4cbb-8495-6f78f2524e13-system-cni-dir\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdzx\" (UniqueName: \"kubernetes.io/projected/a58f9aab-8cec-4b05-979e-7831f7505070-kube-api-access-7mdzx\") pod \"iptables-alerter-qqc5b\" (UID: \"a58f9aab-8cec-4b05-979e-7831f7505070\") " pod="openshift-network-operator/iptables-alerter-qqc5b"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f93f48f-8169-4a8b-885c-b591f433f601-dbus\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059153    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059224    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6df8b67b-5e30-4cbb-8495-6f78f2524e13-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-sysctl-d\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-run\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059343    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-sys\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1e37671-3221-44a1-b4c0-20f45cec8dcf-agent-certs\") pod \"konnectivity-agent-lf995\" (UID: \"c1e37671-3221-44a1-b4c0-20f45cec8dcf\") " pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-socket-dir-parent\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-run-multus-certs\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-slash\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.059580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059519    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-run-ovn\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059546    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-var-lib-openvswitch\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-env-overrides\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psmkm\" (UniqueName: \"kubernetes.io/projected/6cc2ec45-0451-46da-aa90-214fd10d39bd-kube-api-access-psmkm\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9d7824d-7ae5-434d-8ccc-decace1ecd67-hosts-file\") pod \"node-resolver-f4lz4\" (UID: \"e9d7824d-7ae5-434d-8ccc-decace1ecd67\") " pod="openshift-dns/node-resolver-f4lz4"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9d7824d-7ae5-434d-8ccc-decace1ecd67-tmp-dir\") pod \"node-resolver-f4lz4\" (UID: \"e9d7824d-7ae5-434d-8ccc-decace1ecd67\") " pod="openshift-dns/node-resolver-f4lz4"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-var-lib-kubelet\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-ovnkube-config\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059797    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-sys-fs\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2acdf951-0d97-4bb7-bbce-cc70bddb8e47-serviceca\") pod \"node-ca-qfs4z\" (UID: \"2acdf951-0d97-4bb7-bbce-cc70bddb8e47\") " pod="openshift-image-registry/node-ca-qfs4z"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059845    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-cni-dir\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-cni-binary-copy\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059898    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-var-lib-cni-multus\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-cni-bin\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a58f9aab-8cec-4b05-979e-7831f7505070-iptables-alerter-script\") pod \"iptables-alerter-qqc5b\" (UID: \"a58f9aab-8cec-4b05-979e-7831f7505070\") " pod="openshift-network-operator/iptables-alerter-qqc5b"
Sep 17 21:58:25.060457 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.059984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-kubernetes\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.061312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.060006    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.061312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.060028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-var-lib-cni-bin\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.061312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.060050    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-ovn-node-metrics-cert\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.061312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.060073    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a58f9aab-8cec-4b05-979e-7831f7505070-host-slash\") pod \"iptables-alerter-qqc5b\" (UID: \"a58f9aab-8cec-4b05-979e-7831f7505070\") " pod="openshift-network-operator/iptables-alerter-qqc5b"
Sep 17 21:58:25.061312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.060112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-sysctl-conf\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.061312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.060174    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-lib-modules\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.061312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.060194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-system-cni-dir\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.061312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.060217    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-systemd-units\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.061312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.060238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-run-netns\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.061312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.060287    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-run-systemd\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.061312 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.060307    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2acdf951-0d97-4bb7-bbce-cc70bddb8e47-host\") pod \"node-ca-qfs4z\" (UID: \"2acdf951-0d97-4bb7-bbce-cc70bddb8e47\") " pod="openshift-image-registry/node-ca-qfs4z"
Sep 17 21:58:25.077561 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.077536    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cs24v"
Sep 17 21:58:25.082604 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.082584    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cs24v"
Sep 17 21:58:25.160553 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-node-log\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.160553 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6df8b67b-5e30-4cbb-8495-6f78f2524e13-system-cni-dir\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160572    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdzx\" (UniqueName: \"kubernetes.io/projected/a58f9aab-8cec-4b05-979e-7831f7505070-kube-api-access-7mdzx\") pod \"iptables-alerter-qqc5b\" (UID: \"a58f9aab-8cec-4b05-979e-7831f7505070\") " pod="openshift-network-operator/iptables-alerter-qqc5b"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f93f48f-8169-4a8b-885c-b591f433f601-dbus\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6df8b67b-5e30-4cbb-8495-6f78f2524e13-system-cni-dir\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6df8b67b-5e30-4cbb-8495-6f78f2524e13-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-sysctl-d\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160698    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-run\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-sys\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7f93f48f-8169-4a8b-885c-b591f433f601-dbus\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:25.160800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1e37671-3221-44a1-b4c0-20f45cec8dcf-agent-certs\") pod \"konnectivity-agent-lf995\" (UID: \"c1e37671-3221-44a1-b4c0-20f45cec8dcf\") " pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-socket-dir-parent\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-run-multus-certs\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.160864    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-slash\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160929    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-run-ovn\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-sys\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-var-lib-openvswitch\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-env-overrides\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psmkm\" (UniqueName: \"kubernetes.io/projected/6cc2ec45-0451-46da-aa90-214fd10d39bd-kube-api-access-psmkm\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9d7824d-7ae5-434d-8ccc-decace1ecd67-hosts-file\") pod \"node-resolver-f4lz4\" (UID: \"e9d7824d-7ae5-434d-8ccc-decace1ecd67\") " pod="openshift-dns/node-resolver-f4lz4"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161079    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9d7824d-7ae5-434d-8ccc-decace1ecd67-tmp-dir\") pod \"node-resolver-f4lz4\" (UID: \"e9d7824d-7ae5-434d-8ccc-decace1ecd67\") " pod="openshift-dns/node-resolver-f4lz4"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-sysctl-d\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-var-lib-kubelet\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161147    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-var-lib-openvswitch\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161129    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-ovnkube-config\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6df8b67b-5e30-4cbb-8495-6f78f2524e13-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.161424 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-slash\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.162213 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161246    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.162213 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161251    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-run-ovn\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.162213 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-run-multus-certs\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.162213 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.160631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-node-log\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.162213 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161364    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9d7824d-7ae5-434d-8ccc-decace1ecd67-hosts-file\") pod \"node-resolver-f4lz4\" (UID: \"e9d7824d-7ae5-434d-8ccc-decace1ecd67\") " pod="openshift-dns/node-resolver-f4lz4"
Sep 17 21:58:25.162213 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.161446    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs podName:6cc2ec45-0451-46da-aa90-214fd10d39bd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:25.66139137 +0000 UTC m=+4.744979302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs") pod "network-metrics-daemon-n64gh" (UID: "6cc2ec45-0451-46da-aa90-214fd10d39bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:25.162213 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-socket-dir-parent\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.162213 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161534    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 21:58:25.162213 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161546    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9d7824d-7ae5-434d-8ccc-decace1ecd67-tmp-dir\") pod \"node-resolver-f4lz4\" (UID: \"e9d7824d-7ae5-434d-8ccc-decace1ecd67\") " pod="openshift-dns/node-resolver-f4lz4"
Sep 17 21:58:25.162213 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161131    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-run\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.162213 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-env-overrides\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.162712 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.162655    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-sys-fs\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.162712 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.161745    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-var-lib-kubelet\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.163521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.162950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-sys-fs\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.163521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.163080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-ovnkube-config\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.163692 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.163030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2acdf951-0d97-4bb7-bbce-cc70bddb8e47-serviceca\") pod \"node-ca-qfs4z\" (UID: \"2acdf951-0d97-4bb7-bbce-cc70bddb8e47\") " pod="openshift-image-registry/node-ca-qfs4z"
Sep 17 21:58:25.163692 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.163619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-cni-dir\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.163692 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.163631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2acdf951-0d97-4bb7-bbce-cc70bddb8e47-serviceca\") pod \"node-ca-qfs4z\" (UID: \"2acdf951-0d97-4bb7-bbce-cc70bddb8e47\") " pod="openshift-image-registry/node-ca-qfs4z"
Sep 17 21:58:25.163692 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.163675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-cni-binary-copy\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.163901 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.163710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-var-lib-cni-multus\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.163901 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.163746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-cni-dir\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.163901 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.163746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-cni-bin\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.163901 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.163798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a58f9aab-8cec-4b05-979e-7831f7505070-iptables-alerter-script\") pod \"iptables-alerter-qqc5b\" (UID: \"a58f9aab-8cec-4b05-979e-7831f7505070\") " pod="openshift-network-operator/iptables-alerter-qqc5b"
Sep 17 21:58:25.163901 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.163805    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-cni-bin\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.164425 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.164379    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a58f9aab-8cec-4b05-979e-7831f7505070-iptables-alerter-script\") pod \"iptables-alerter-qqc5b\" (UID: \"a58f9aab-8cec-4b05-979e-7831f7505070\") " pod="openshift-network-operator/iptables-alerter-qqc5b"
Sep 17 21:58:25.164521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.164429    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-var-lib-cni-multus\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.164521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.164431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-cni-binary-copy\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.164521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.164451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-kubernetes\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.164521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.164513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.164719 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.164543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-var-lib-cni-bin\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.164719 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.164557    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-kubernetes\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.164719 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.164593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-var-lib-cni-bin\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.164719 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.164621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.164719 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.164676    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-ovn-node-metrics-cert\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.165544 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.165517    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c1e37671-3221-44a1-b4c0-20f45cec8dcf-agent-certs\") pod \"konnectivity-agent-lf995\" (UID: \"c1e37671-3221-44a1-b4c0-20f45cec8dcf\") " pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:25.166434 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166269    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a58f9aab-8cec-4b05-979e-7831f7505070-host-slash\") pod \"iptables-alerter-qqc5b\" (UID: \"a58f9aab-8cec-4b05-979e-7831f7505070\") " pod="openshift-network-operator/iptables-alerter-qqc5b"
Sep 17 21:58:25.166434 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-sysctl-conf\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.166434 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a58f9aab-8cec-4b05-979e-7831f7505070-host-slash\") pod \"iptables-alerter-qqc5b\" (UID: \"a58f9aab-8cec-4b05-979e-7831f7505070\") " pod="openshift-network-operator/iptables-alerter-qqc5b"
Sep 17 21:58:25.166434 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166414    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-sysctl-conf\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.166434 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-lib-modules\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.166717 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-system-cni-dir\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.166717 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166492    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-systemd-units\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.166717 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166516    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-lib-modules\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.166717 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-run-netns\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.166717 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-system-cni-dir\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.166717 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-run-netns\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.166717 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-systemd-units\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.166717 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-run-systemd\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.166717 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2acdf951-0d97-4bb7-bbce-cc70bddb8e47-host\") pod \"node-ca-qfs4z\" (UID: \"2acdf951-0d97-4bb7-bbce-cc70bddb8e47\") " pod="openshift-image-registry/node-ca-qfs4z"
Sep 17 21:58:25.166717 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bhhh\" (UniqueName: \"kubernetes.io/projected/2acdf951-0d97-4bb7-bbce-cc70bddb8e47-kube-api-access-4bhhh\") pod \"node-ca-qfs4z\" (UID: \"2acdf951-0d97-4bb7-bbce-cc70bddb8e47\") " pod="openshift-image-registry/node-ca-qfs4z"
Sep 17 21:58:25.166717 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-run-systemd\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.167195 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.166981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfrqx\" (UniqueName: \"kubernetes.io/projected/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-kube-api-access-jfrqx\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.167195 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167023    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6df8b67b-5e30-4cbb-8495-6f78f2524e13-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.167195 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167053    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-systemd\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.167195 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-device-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.167195 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-conf-dir\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.167445 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2acdf951-0d97-4bb7-bbce-cc70bddb8e47-host\") pod \"node-ca-qfs4z\" (UID: \"2acdf951-0d97-4bb7-bbce-cc70bddb8e47\") " pod="openshift-image-registry/node-ca-qfs4z"
Sep 17 21:58:25.167445 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167269    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-conf-dir\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.167445 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6df8b67b-5e30-4cbb-8495-6f78f2524e13-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.167445 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-tuned\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.167626 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167461    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-ovn-node-metrics-cert\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.167626 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-systemd\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.167889 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167845    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-device-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.167969 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.167860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-run-k8s-cni-cncf-io\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.168299 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-run-k8s-cni-cncf-io\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.168407 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-run-netns\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.168407 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-host-run-netns\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.168661 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-kubelet\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.168661 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-etc-openvswitch\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.168661 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168507    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-kubelet\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.168661 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168524    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-etc-openvswitch\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.168661 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-log-socket\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.168661 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168564    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-log-socket\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.168661 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168572    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6df8b67b-5e30-4cbb-8495-6f78f2524e13-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.168661 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-var-lib-kubelet\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.168661 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a921b3aa-f930-4ac3-9cb4-9af843a381df-tmp\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.168661 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74xkq\" (UniqueName: \"kubernetes.io/projected/a921b3aa-f930-4ac3-9cb4-9af843a381df-kube-api-access-74xkq\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-var-lib-kubelet\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-cni-netd\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168818    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-cni-netd\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168830    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168845    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784fz\" (UniqueName: \"kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz\") pod \"network-check-target-q8xvj\" (UID: \"032d58e2-2e31-40c8-87cc-a4d5939db410\") " pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6df8b67b-5e30-4cbb-8495-6f78f2524e13-cni-binary-copy\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.168892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f93f48f-8169-4a8b-885c-b591f433f601-kubelet-config\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.168940    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.169019    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret podName:7f93f48f-8169-4a8b-885c-b591f433f601 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:25.668999327 +0000 UTC m=+4.752587256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret") pod "global-pull-secret-syncer-rffj6" (UID: "7f93f48f-8169-4a8b-885c-b591f433f601") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-registration-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169073    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-etc-kubernetes\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-ovnkube-script-lib\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1e37671-3221-44a1-b4c0-20f45cec8dcf-konnectivity-ca\") pod \"konnectivity-agent-lf995\" (UID: \"c1e37671-3221-44a1-b4c0-20f45cec8dcf\") " pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6df8b67b-5e30-4cbb-8495-6f78f2524e13-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.170578 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-run-openvswitch\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdz7g\" (UniqueName: \"kubernetes.io/projected/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-kube-api-access-sdz7g\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-etc-kubernetes\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7f93f48f-8169-4a8b-885c-b591f433f601-kubelet-config\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kmwt\" (UniqueName: \"kubernetes.io/projected/6df8b67b-5e30-4cbb-8495-6f78f2524e13-kube-api-access-6kmwt\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-run-openvswitch\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169306    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6df8b67b-5e30-4cbb-8495-6f78f2524e13-cni-binary-copy\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-modprobe-d\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-host\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-socket-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrvnx\" (UniqueName: \"kubernetes.io/projected/425aa0ec-46a3-4712-9a04-91b742dde2b6-kube-api-access-lrvnx\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-hostroot\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169611    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6df8b67b-5e30-4cbb-8495-6f78f2524e13-cnibin\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-sysconfig\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxtjq\" (UniqueName: \"kubernetes.io/projected/e9d7824d-7ae5-434d-8ccc-decace1ecd67-kube-api-access-wxtjq\") pod \"node-resolver-f4lz4\" (UID: \"e9d7824d-7ae5-434d-8ccc-decace1ecd67\") " pod="openshift-dns/node-resolver-f4lz4"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-cnibin\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-daemon-config\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.171420 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169743    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-socket-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/425aa0ec-46a3-4712-9a04-91b742dde2b6-registration-dir\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169756    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-ovnkube-script-lib\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169745    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c1e37671-3221-44a1-b4c0-20f45cec8dcf-konnectivity-ca\") pod \"konnectivity-agent-lf995\" (UID: \"c1e37671-3221-44a1-b4c0-20f45cec8dcf\") " pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-sysconfig\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6df8b67b-5e30-4cbb-8495-6f78f2524e13-cnibin\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169846    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-modprobe-d\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-hostroot\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169885    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a921b3aa-f930-4ac3-9cb4-9af843a381df-host\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.169957    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-cnibin\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.170327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-multus-daemon-config\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.170776    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a921b3aa-f930-4ac3-9cb4-9af843a381df-etc-tuned\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.171879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdzx\" (UniqueName: \"kubernetes.io/projected/a58f9aab-8cec-4b05-979e-7831f7505070-kube-api-access-7mdzx\") pod \"iptables-alerter-qqc5b\" (UID: \"a58f9aab-8cec-4b05-979e-7831f7505070\") " pod="openshift-network-operator/iptables-alerter-qqc5b"
Sep 17 21:58:25.172292 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.172088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a921b3aa-f930-4ac3-9cb4-9af843a381df-tmp\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.172835 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.172742    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psmkm\" (UniqueName: \"kubernetes.io/projected/6cc2ec45-0451-46da-aa90-214fd10d39bd-kube-api-access-psmkm\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:25.174910 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.174856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bhhh\" (UniqueName: \"kubernetes.io/projected/2acdf951-0d97-4bb7-bbce-cc70bddb8e47-kube-api-access-4bhhh\") pod \"node-ca-qfs4z\" (UID: \"2acdf951-0d97-4bb7-bbce-cc70bddb8e47\") " pod="openshift-image-registry/node-ca-qfs4z"
Sep 17 21:58:25.175343 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.175323    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:25.175444 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.175351    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:25.175444 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.175364    2583 projected.go:196] Error preparing data for projected volume kube-api-access-784fz for pod openshift-network-diagnostics/network-check-target-q8xvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:25.175444 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.175367    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfrqx\" (UniqueName: \"kubernetes.io/projected/2c8fb3d3-7b60-4cc8-92d7-067809b20b92-kube-api-access-jfrqx\") pod \"multus-2n6n9\" (UID: \"2c8fb3d3-7b60-4cc8-92d7-067809b20b92\") " pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.175444 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.175417    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz podName:032d58e2-2e31-40c8-87cc-a4d5939db410 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:25.675405326 +0000 UTC m=+4.758993257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-784fz" (UniqueName: "kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz") pod "network-check-target-q8xvj" (UID: "032d58e2-2e31-40c8-87cc-a4d5939db410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:25.175770 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.175752    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74xkq\" (UniqueName: \"kubernetes.io/projected/a921b3aa-f930-4ac3-9cb4-9af843a381df-kube-api-access-74xkq\") pod \"tuned-wcphq\" (UID: \"a921b3aa-f930-4ac3-9cb4-9af843a381df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.177539 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.177516    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kmwt\" (UniqueName: \"kubernetes.io/projected/6df8b67b-5e30-4cbb-8495-6f78f2524e13-kube-api-access-6kmwt\") pod \"multus-additional-cni-plugins-wlb9z\" (UID: \"6df8b67b-5e30-4cbb-8495-6f78f2524e13\") " pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.178078 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.178050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxtjq\" (UniqueName: \"kubernetes.io/projected/e9d7824d-7ae5-434d-8ccc-decace1ecd67-kube-api-access-wxtjq\") pod \"node-resolver-f4lz4\" (UID: \"e9d7824d-7ae5-434d-8ccc-decace1ecd67\") " pod="openshift-dns/node-resolver-f4lz4"
Sep 17 21:58:25.178388 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.178370    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdz7g\" (UniqueName: \"kubernetes.io/projected/628981f9-9f9e-4bb1-9eb1-fd1b0241989f-kube-api-access-sdz7g\") pod \"ovnkube-node-vpmct\" (UID: \"628981f9-9f9e-4bb1-9eb1-fd1b0241989f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.178489 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.178468    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrvnx\" (UniqueName: \"kubernetes.io/projected/425aa0ec-46a3-4712-9a04-91b742dde2b6-kube-api-access-lrvnx\") pod \"aws-ebs-csi-driver-node-6wxxs\" (UID: \"425aa0ec-46a3-4712-9a04-91b742dde2b6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.184792 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.184756    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal" event={"ID":"ca36c8d3176c4f0a591a991efcdaef22","Type":"ContainerStarted","Data":"8ffee43ca49aca01361bba4f4253494c29714b85de0af41b1421fe56fdacc9e7"}
Sep 17 21:58:25.187594 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.187572    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal" event={"ID":"6a6232eedc32fbab34ed57bf917cc434","Type":"ContainerStarted","Data":"fb52d7b3bf9d588023afa6e84b82690e86631899c28a315aa0b73b0741aa37b5"}
Sep 17 21:58:25.322265 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.322234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wlb9z"
Sep 17 21:58:25.330385 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.330362    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qqc5b"
Sep 17 21:58:25.343219 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.343191    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:25.347806 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.347786    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wcphq"
Sep 17 21:58:25.355214 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.355196    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f4lz4"
Sep 17 21:58:25.362919 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.362901    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qfs4z"
Sep 17 21:58:25.370483 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.370462    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2n6n9"
Sep 17 21:58:25.379033 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.379011    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:25.382844 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.382828    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs"
Sep 17 21:58:25.463260 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.463173    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:25.672503 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.672468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:25.672683 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.672533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:25.672683 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.672632    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:25.672683 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.672669    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:25.672801 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.672705    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret podName:7f93f48f-8169-4a8b-885c-b591f433f601 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:26.672689039 +0000 UTC m=+5.756276971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret") pod "global-pull-secret-syncer-rffj6" (UID: "7f93f48f-8169-4a8b-885c-b591f433f601") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:25.672801 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.672721    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs podName:6cc2ec45-0451-46da-aa90-214fd10d39bd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:26.672714153 +0000 UTC m=+5.756302081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs") pod "network-metrics-daemon-n64gh" (UID: "6cc2ec45-0451-46da-aa90-214fd10d39bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:25.773220 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:25.773131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784fz\" (UniqueName: \"kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz\") pod \"network-check-target-q8xvj\" (UID: \"032d58e2-2e31-40c8-87cc-a4d5939db410\") " pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:25.773354 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.773294    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:25.773354 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.773320    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:25.773354 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.773329    2583 projected.go:196] Error preparing data for projected volume kube-api-access-784fz for pod openshift-network-diagnostics/network-check-target-q8xvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:25.773450 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:25.773393    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz podName:032d58e2-2e31-40c8-87cc-a4d5939db410 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:26.773378938 +0000 UTC m=+5.856966865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-784fz" (UniqueName: "kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz") pod "network-check-target-q8xvj" (UID: "032d58e2-2e31-40c8-87cc-a4d5939db410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:26.086033 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.085931    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:25 +0000 UTC" deadline="2028-03-18 09:55:04.458270216 +0000 UTC"
Sep 17 21:58:26.086033 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.085991    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13139h56m38.372284919s"
Sep 17 21:58:26.113474 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:26.113428    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d7824d_7ae5_434d_8ccc_decace1ecd67.slice/crio-cc26e804bf67720b2c2d1af13c3b216f063c0f44c3bcd502eb26706a870160c2 WatchSource:0}: Error finding container cc26e804bf67720b2c2d1af13c3b216f063c0f44c3bcd502eb26706a870160c2: Status 404 returned error can't find the container with id cc26e804bf67720b2c2d1af13c3b216f063c0f44c3bcd502eb26706a870160c2
Sep 17 21:58:26.114440 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:26.114416    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df8b67b_5e30_4cbb_8495_6f78f2524e13.slice/crio-8081bb87f23af2673e832410c1c1a838fdc7591a756f9bd965b94275dddcfd1e WatchSource:0}: Error finding container 8081bb87f23af2673e832410c1c1a838fdc7591a756f9bd965b94275dddcfd1e: Status 404 returned error can't find the container with id 8081bb87f23af2673e832410c1c1a838fdc7591a756f9bd965b94275dddcfd1e
Sep 17 21:58:26.115919 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:26.115896    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628981f9_9f9e_4bb1_9eb1_fd1b0241989f.slice/crio-86472ecf0929dfbbf7be4c76a2dd28a09b563fff865da08ef4d0297540ac029e WatchSource:0}: Error finding container 86472ecf0929dfbbf7be4c76a2dd28a09b563fff865da08ef4d0297540ac029e: Status 404 returned error can't find the container with id 86472ecf0929dfbbf7be4c76a2dd28a09b563fff865da08ef4d0297540ac029e
Sep 17 21:58:26.119449 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:26.119428    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425aa0ec_46a3_4712_9a04_91b742dde2b6.slice/crio-545b9c4d25611490a93d250ed0b7131b657e6139ac265485ffe9269bb8550370 WatchSource:0}: Error finding container 545b9c4d25611490a93d250ed0b7131b657e6139ac265485ffe9269bb8550370: Status 404 returned error can't find the container with id 545b9c4d25611490a93d250ed0b7131b657e6139ac265485ffe9269bb8550370
Sep 17 21:58:26.120980 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:26.120959    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8fb3d3_7b60_4cc8_92d7_067809b20b92.slice/crio-755dd4bb68546af07bdb83393ae2f342e4cf56d4956dbc24b3c3511882853179 WatchSource:0}: Error finding container 755dd4bb68546af07bdb83393ae2f342e4cf56d4956dbc24b3c3511882853179: Status 404 returned error can't find the container with id 755dd4bb68546af07bdb83393ae2f342e4cf56d4956dbc24b3c3511882853179
Sep 17 21:58:26.122264 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:26.122238    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e37671_3221_44a1_b4c0_20f45cec8dcf.slice/crio-a9092cbb64e25a3348d320806cb201f04fa211d897d5ef556658bfcbeececd7a WatchSource:0}: Error finding container a9092cbb64e25a3348d320806cb201f04fa211d897d5ef556658bfcbeececd7a: Status 404 returned error can't find the container with id a9092cbb64e25a3348d320806cb201f04fa211d897d5ef556658bfcbeececd7a
Sep 17 21:58:26.123701 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:26.123671    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda921b3aa_f930_4ac3_9cb4_9af843a381df.slice/crio-d481d5c8b2e19ab148f6ca7925fb732af13cf0d8dab8ae200624f2dbf1b9326d WatchSource:0}: Error finding container d481d5c8b2e19ab148f6ca7925fb732af13cf0d8dab8ae200624f2dbf1b9326d: Status 404 returned error can't find the container with id d481d5c8b2e19ab148f6ca7925fb732af13cf0d8dab8ae200624f2dbf1b9326d
Sep 17 21:58:26.192693 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.192661    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qqc5b" event={"ID":"a58f9aab-8cec-4b05-979e-7831f7505070","Type":"ContainerStarted","Data":"3784c4ff2c76945e93bf0e051093d0c1f3610fcbc0ebd4e6586849d7243f9241"}
Sep 17 21:58:26.195564 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.195532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qfs4z" event={"ID":"2acdf951-0d97-4bb7-bbce-cc70bddb8e47","Type":"ContainerStarted","Data":"0013e5982522f5f33e4b9a7fe89e6cf2e4e064f98c8fd69fb45e4159e5eecd24"}
Sep 17 21:58:26.197993 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.197965    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lf995" event={"ID":"c1e37671-3221-44a1-b4c0-20f45cec8dcf","Type":"ContainerStarted","Data":"a9092cbb64e25a3348d320806cb201f04fa211d897d5ef556658bfcbeececd7a"}
Sep 17 21:58:26.199085 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.199063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2n6n9" event={"ID":"2c8fb3d3-7b60-4cc8-92d7-067809b20b92","Type":"ContainerStarted","Data":"755dd4bb68546af07bdb83393ae2f342e4cf56d4956dbc24b3c3511882853179"}
Sep 17 21:58:26.203074 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.203046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct" event={"ID":"628981f9-9f9e-4bb1-9eb1-fd1b0241989f","Type":"ContainerStarted","Data":"86472ecf0929dfbbf7be4c76a2dd28a09b563fff865da08ef4d0297540ac029e"}
Sep 17 21:58:26.207185 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.207163    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wcphq" event={"ID":"a921b3aa-f930-4ac3-9cb4-9af843a381df","Type":"ContainerStarted","Data":"d481d5c8b2e19ab148f6ca7925fb732af13cf0d8dab8ae200624f2dbf1b9326d"}
Sep 17 21:58:26.210063 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.210044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs" event={"ID":"425aa0ec-46a3-4712-9a04-91b742dde2b6","Type":"ContainerStarted","Data":"545b9c4d25611490a93d250ed0b7131b657e6139ac265485ffe9269bb8550370"}
Sep 17 21:58:26.212916 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.212894    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlb9z" event={"ID":"6df8b67b-5e30-4cbb-8495-6f78f2524e13","Type":"ContainerStarted","Data":"8081bb87f23af2673e832410c1c1a838fdc7591a756f9bd965b94275dddcfd1e"}
Sep 17 21:58:26.214887 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.214867    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f4lz4" event={"ID":"e9d7824d-7ae5-434d-8ccc-decace1ecd67","Type":"ContainerStarted","Data":"cc26e804bf67720b2c2d1af13c3b216f063c0f44c3bcd502eb26706a870160c2"}
Sep 17 21:58:26.679125 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.679086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:26.679335 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.679178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:26.679335 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:26.679288    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:26.679450 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:26.679349    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs podName:6cc2ec45-0451-46da-aa90-214fd10d39bd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:28.679330757 +0000 UTC m=+7.762918691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs") pod "network-metrics-daemon-n64gh" (UID: "6cc2ec45-0451-46da-aa90-214fd10d39bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:26.679684 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:26.679288    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:26.679776 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:26.679727    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret podName:7f93f48f-8169-4a8b-885c-b591f433f601 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:28.679709809 +0000 UTC m=+7.763297740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret") pod "global-pull-secret-syncer-rffj6" (UID: "7f93f48f-8169-4a8b-885c-b591f433f601") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:26.780702 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:26.780646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784fz\" (UniqueName: \"kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz\") pod \"network-check-target-q8xvj\" (UID: \"032d58e2-2e31-40c8-87cc-a4d5939db410\") " pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:26.780955 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:26.780937    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:26.781012 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:26.780965    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:26.781012 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:26.780979    2583 projected.go:196] Error preparing data for projected volume kube-api-access-784fz for pod openshift-network-diagnostics/network-check-target-q8xvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:26.781128 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:26.781039    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz podName:032d58e2-2e31-40c8-87cc-a4d5939db410 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:28.781019945 +0000 UTC m=+7.864607887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-784fz" (UniqueName: "kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz") pod "network-check-target-q8xvj" (UID: "032d58e2-2e31-40c8-87cc-a4d5939db410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:27.086609 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:27.086498    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 21:53:25 +0000 UTC" deadline="2028-06-30 23:44:14.016278584 +0000 UTC"
Sep 17 21:58:27.086609 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:27.086559    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15649h45m46.92972305s"
Sep 17 21:58:27.156817 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:27.156782    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 21:58:27.182050 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:27.181256    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:27.182050 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:27.181389    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:27.182050 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:27.181788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:27.182050 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:27.181875    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:27.182050 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:27.181938    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:27.182050 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:27.182012    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:27.237290 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:27.236902    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a6232eedc32fbab34ed57bf917cc434" containerID="2ed99aef1cd249dd022d18be3c4cccc1beaede2a998477e387cdc0111124cae8" exitCode=0
Sep 17 21:58:27.237290 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:27.237009    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal" event={"ID":"6a6232eedc32fbab34ed57bf917cc434","Type":"ContainerDied","Data":"2ed99aef1cd249dd022d18be3c4cccc1beaede2a998477e387cdc0111124cae8"}
Sep 17 21:58:27.264478 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:27.263658    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal" event={"ID":"ca36c8d3176c4f0a591a991efcdaef22","Type":"ContainerStarted","Data":"ab62b824872e65ee652953171fdaf48777c3ed8a9856cdceb204f5ade6f30f51"}
Sep 17 21:58:27.280025 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:27.278945    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal" podStartSLOduration=3.27892491 podStartE2EDuration="3.27892491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:27.278736005 +0000 UTC m=+6.362323957" watchObservedRunningTime="2026-09-17 21:58:27.27892491 +0000 UTC m=+6.362512860"
Sep 17 21:58:28.293268 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:28.293199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal" event={"ID":"6a6232eedc32fbab34ed57bf917cc434","Type":"ContainerStarted","Data":"5746500848e0fe18524fe268a34669a5f517ecf30b35aa8cb23b829a87f56e42"}
Sep 17 21:58:28.698428 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:28.697628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:28.698428 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:28.697706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:28.698428 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:28.697869    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:28.698428 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:28.697935    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs podName:6cc2ec45-0451-46da-aa90-214fd10d39bd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:32.697915252 +0000 UTC m=+11.781503206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs") pod "network-metrics-daemon-n64gh" (UID: "6cc2ec45-0451-46da-aa90-214fd10d39bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:28.698428 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:28.698340    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:28.698428 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:28.698391    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret podName:7f93f48f-8169-4a8b-885c-b591f433f601 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:32.698376282 +0000 UTC m=+11.781964248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret") pod "global-pull-secret-syncer-rffj6" (UID: "7f93f48f-8169-4a8b-885c-b591f433f601") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:28.798252 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:28.798213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784fz\" (UniqueName: \"kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz\") pod \"network-check-target-q8xvj\" (UID: \"032d58e2-2e31-40c8-87cc-a4d5939db410\") " pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:28.798491 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:28.798466    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:28.798563 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:28.798512    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:28.798563 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:28.798527    2583 projected.go:196] Error preparing data for projected volume kube-api-access-784fz for pod openshift-network-diagnostics/network-check-target-q8xvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:28.798685 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:28.798602    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz podName:032d58e2-2e31-40c8-87cc-a4d5939db410 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:32.798582615 +0000 UTC m=+11.882170548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-784fz" (UniqueName: "kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz") pod "network-check-target-q8xvj" (UID: "032d58e2-2e31-40c8-87cc-a4d5939db410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:29.180702 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:29.180608    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:29.180702 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:29.180651    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:29.180953 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:29.180757    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:29.181216 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:29.181183    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:29.181321 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:29.181242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:29.181382 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:29.181318    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:31.180893 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:31.180854    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:31.181477 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:31.181002    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:31.181477 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:31.180858    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:31.181477 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:31.181163    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:31.181477 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:31.181234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:31.181477 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:31.181308    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:32.733901 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:32.732960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:32.733901 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:32.733049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:32.733901 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:32.733241    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:32.733901 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:32.733308    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs podName:6cc2ec45-0451-46da-aa90-214fd10d39bd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:40.733287785 +0000 UTC m=+19.816875718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs") pod "network-metrics-daemon-n64gh" (UID: "6cc2ec45-0451-46da-aa90-214fd10d39bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:32.733901 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:32.733814    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:32.733901 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:32.733865    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret podName:7f93f48f-8169-4a8b-885c-b591f433f601 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:40.733849081 +0000 UTC m=+19.817437022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret") pod "global-pull-secret-syncer-rffj6" (UID: "7f93f48f-8169-4a8b-885c-b591f433f601") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:32.834005 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:32.833967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784fz\" (UniqueName: \"kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz\") pod \"network-check-target-q8xvj\" (UID: \"032d58e2-2e31-40c8-87cc-a4d5939db410\") " pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:32.834233 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:32.834177    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:32.834233 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:32.834200    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:32.834233 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:32.834210    2583 projected.go:196] Error preparing data for projected volume kube-api-access-784fz for pod openshift-network-diagnostics/network-check-target-q8xvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:32.834401 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:32.834267    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz podName:032d58e2-2e31-40c8-87cc-a4d5939db410 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:40.834247436 +0000 UTC m=+19.917835410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-784fz" (UniqueName: "kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz") pod "network-check-target-q8xvj" (UID: "032d58e2-2e31-40c8-87cc-a4d5939db410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:33.182212 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:33.181498    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:33.182212 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:33.181625    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:33.182212 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:33.181980    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:33.182212 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:33.182066    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:33.182212 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:33.182118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:33.182758 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:33.182729    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:35.180789 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:35.180746    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:35.181259 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:35.180798    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:35.181259 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:35.180747    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:35.181259 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:35.180891    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:35.181259 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:35.180996    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:35.181259 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:35.181100    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:37.179988 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:37.179929    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:37.179988 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:37.179956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:37.179988 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:37.179933    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:37.180562 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:37.180082    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:37.180562 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:37.180164    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:37.180562 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:37.180214    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:39.180651 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:39.180611    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:39.181069 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:39.180657    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:39.181069 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:39.180920    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:39.181240 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:39.181186    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:39.181370 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:39.181275    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:39.181426 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:39.181404    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:40.792711 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:40.792672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:40.793185 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:40.792755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:40.793185 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:40.792849    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:40.793185 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:40.792878    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:40.793185 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:40.792921    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs podName:6cc2ec45-0451-46da-aa90-214fd10d39bd nodeName:}" failed. No retries permitted until 2026-09-17 21:58:56.792900307 +0000 UTC m=+35.876488250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs") pod "network-metrics-daemon-n64gh" (UID: "6cc2ec45-0451-46da-aa90-214fd10d39bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:40.793185 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:40.792941    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret podName:7f93f48f-8169-4a8b-885c-b591f433f601 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:56.792930758 +0000 UTC m=+35.876518688 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret") pod "global-pull-secret-syncer-rffj6" (UID: "7f93f48f-8169-4a8b-885c-b591f433f601") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:40.893972 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:40.893943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784fz\" (UniqueName: \"kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz\") pod \"network-check-target-q8xvj\" (UID: \"032d58e2-2e31-40c8-87cc-a4d5939db410\") " pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:40.894151 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:40.894122    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:40.894190 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:40.894157    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:40.894190 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:40.894167    2583 projected.go:196] Error preparing data for projected volume kube-api-access-784fz for pod openshift-network-diagnostics/network-check-target-q8xvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:40.894260 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:40.894219    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz podName:032d58e2-2e31-40c8-87cc-a4d5939db410 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:56.894204832 +0000 UTC m=+35.977792765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-784fz" (UniqueName: "kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz") pod "network-check-target-q8xvj" (UID: "032d58e2-2e31-40c8-87cc-a4d5939db410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:41.180391 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:41.180353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:41.180577 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:41.180353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:41.180577 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:41.180489    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:41.180577 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:41.180353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:41.180726 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:41.180585    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:41.180726 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:41.180641    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:43.180693 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:43.180666    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:43.181180 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:43.180749    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:43.181180 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:43.180824    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:43.181180 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:43.180915    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:43.181180 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:43.180948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:43.181180 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:43.181005    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:44.324510 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.324178    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wcphq" event={"ID":"a921b3aa-f930-4ac3-9cb4-9af843a381df","Type":"ContainerStarted","Data":"381e75c117cde4ce8cda45fbc438c89261b17e0d58efb105c0c7ed78256d04ab"}
Sep 17 21:58:44.325702 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.325684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs" event={"ID":"425aa0ec-46a3-4712-9a04-91b742dde2b6","Type":"ContainerStarted","Data":"d2b0fea78437b4a923ee97dfd5a1a2be615d3c8a4202de990f2e409f8bc14c3b"}
Sep 17 21:58:44.327074 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.327053    2583 generic.go:356] "Generic (PLEG): container finished" podID="6df8b67b-5e30-4cbb-8495-6f78f2524e13" containerID="6786aa47bdce7fb07c5d8b2c5ea253197a7f09da300a7cdbc617dfc9991e8061" exitCode=0
Sep 17 21:58:44.327177 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.327117    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlb9z" event={"ID":"6df8b67b-5e30-4cbb-8495-6f78f2524e13","Type":"ContainerDied","Data":"6786aa47bdce7fb07c5d8b2c5ea253197a7f09da300a7cdbc617dfc9991e8061"}
Sep 17 21:58:44.328625 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.328581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f4lz4" event={"ID":"e9d7824d-7ae5-434d-8ccc-decace1ecd67","Type":"ContainerStarted","Data":"d50d6358e2fcc6e6d21e605358f96a2bc090a1c36021df105dd4eac9ec33b258"}
Sep 17 21:58:44.329947 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.329927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qfs4z" event={"ID":"2acdf951-0d97-4bb7-bbce-cc70bddb8e47","Type":"ContainerStarted","Data":"51568c30fccedc8829fb4943c550e2b7ab9f5687d1c21ee9915fca38508e7226"}
Sep 17 21:58:44.331309 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.331283    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lf995" event={"ID":"c1e37671-3221-44a1-b4c0-20f45cec8dcf","Type":"ContainerStarted","Data":"23933957dd3aaea3c60b344cd3a21cac2fecd212d5c51d41c137381be25a61d6"}
Sep 17 21:58:44.332774 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.332754    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2n6n9" event={"ID":"2c8fb3d3-7b60-4cc8-92d7-067809b20b92","Type":"ContainerStarted","Data":"0e23272ba6532a137b09219877e1871f4b4273a02a6f859255e11e36f821bd74"}
Sep 17 21:58:44.335569 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.335554    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 21:58:44.335882 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.335863    2583 generic.go:356] "Generic (PLEG): container finished" podID="628981f9-9f9e-4bb1-9eb1-fd1b0241989f" containerID="24476b0227789387b38ca0a969171b10e6896b28046396f6f8b08ea9abe021b1" exitCode=1
Sep 17 21:58:44.335936 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.335895    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct" event={"ID":"628981f9-9f9e-4bb1-9eb1-fd1b0241989f","Type":"ContainerStarted","Data":"185ab1ba8f444b30712eb6ba79118b609a925ba2e144b048877aa85235a82c57"}
Sep 17 21:58:44.335980 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.335966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct" event={"ID":"628981f9-9f9e-4bb1-9eb1-fd1b0241989f","Type":"ContainerStarted","Data":"9712b16344301eeb2b34f5ab9e56e69bdf612fee201a99def1c20cef9098c7e3"}
Sep 17 21:58:44.336028 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.336017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct" event={"ID":"628981f9-9f9e-4bb1-9eb1-fd1b0241989f","Type":"ContainerStarted","Data":"f462a0e992bef3bff02ea299f1bd8ba76fd4e218c4f622fe764da88b0d062e2a"}
Sep 17 21:58:44.336080 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.336068    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct" event={"ID":"628981f9-9f9e-4bb1-9eb1-fd1b0241989f","Type":"ContainerStarted","Data":"62b0c4c868acd23af5b85ed99bfde4149823245caf33b6d684a8ddbee8e4b40b"}
Sep 17 21:58:44.336130 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.336118    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct" event={"ID":"628981f9-9f9e-4bb1-9eb1-fd1b0241989f","Type":"ContainerDied","Data":"24476b0227789387b38ca0a969171b10e6896b28046396f6f8b08ea9abe021b1"}
Sep 17 21:58:44.336204 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.336190    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct" event={"ID":"628981f9-9f9e-4bb1-9eb1-fd1b0241989f","Type":"ContainerStarted","Data":"863d9a205d7268b9644e6bec429e4bdfa89ff8209d70887fa36e869b65f92457"}
Sep 17 21:58:44.341828 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.341778    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal" podStartSLOduration=20.341759347 podStartE2EDuration="20.341759347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:28.309993855 +0000 UTC m=+7.393581831" watchObservedRunningTime="2026-09-17 21:58:44.341759347 +0000 UTC m=+23.425347299"
Sep 17 21:58:44.342370 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.342332    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wcphq" podStartSLOduration=3.9626576890000003 podStartE2EDuration="21.342321849s" totalImagesPullingTime="17.37966416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:44.341155827 +0000 UTC m=+23.424743798" watchObservedRunningTime="2026-09-17 21:58:44.342321849 +0000 UTC m=+23.425909799"
Sep 17 21:58:44.373995 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.373948    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lf995" podStartSLOduration=8.250054831 podStartE2EDuration="21.373932932s" totalImagesPullingTime="13.123878101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:44.373747645 +0000 UTC m=+23.457335611" watchObservedRunningTime="2026-09-17 21:58:44.373932932 +0000 UTC m=+23.457520880"
Sep 17 21:58:44.406865 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.406826    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f4lz4" podStartSLOduration=4.043253552 podStartE2EDuration="21.406810358s" totalImagesPullingTime="17.363556806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:44.392238087 +0000 UTC m=+23.475826037" watchObservedRunningTime="2026-09-17 21:58:44.406810358 +0000 UTC m=+23.490398307"
Sep 17 21:58:44.406998 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.406892    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qfs4z" podStartSLOduration=4.026216589 podStartE2EDuration="21.406887662s" totalImagesPullingTime="17.380671073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:44.406258588 +0000 UTC m=+23.489846561" watchObservedRunningTime="2026-09-17 21:58:44.406887662 +0000 UTC m=+23.490475611"
Sep 17 21:58:44.425204 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.425160    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2n6n9" podStartSLOduration=3.991997797 podStartE2EDuration="21.425129485s" totalImagesPullingTime="17.433131688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:44.425021998 +0000 UTC m=+23.508609969" watchObservedRunningTime="2026-09-17 21:58:44.425129485 +0000 UTC m=+23.508717434"
Sep 17 21:58:44.955400 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:44.955255    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 21:58:45.146781 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:45.146682    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T21:58:44.95539571Z","UUID":"aa90b6ce-ba02-468a-918e-97187063b14b","Handler":null,"Name":"","Endpoint":""}
Sep 17 21:58:45.148239 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:45.148216    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 21:58:45.148359 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:45.148251    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 21:58:45.180066 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:45.180037    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:45.180066 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:45.180043    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:45.180254 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:45.180202    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:45.180254 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:45.180244    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:45.180366 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:45.180346    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:45.180446 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:45.180427    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:45.340568 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:45.340529    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qqc5b" event={"ID":"a58f9aab-8cec-4b05-979e-7831f7505070","Type":"ContainerStarted","Data":"f449de473e142a69316f6aa289eb10b0a75a9c7a7135b878342dee2610b46778"}
Sep 17 21:58:45.342762 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:45.342732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs" event={"ID":"425aa0ec-46a3-4712-9a04-91b742dde2b6","Type":"ContainerStarted","Data":"e24e2434767ca8b861b910fc89d0f31640a65b53dee2dd54f2634ff02e4a15f8"}
Sep 17 21:58:45.355414 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:45.355363    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qqc5b" podStartSLOduration=4.9726450490000005 podStartE2EDuration="22.355344486s" totalImagesPullingTime="17.382699437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:45.354812071 +0000 UTC m=+24.438400046" watchObservedRunningTime="2026-09-17 21:58:45.355344486 +0000 UTC m=+24.438932438"
Sep 17 21:58:46.349583 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:46.349561    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 21:58:46.350119 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:46.350084    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct" event={"ID":"628981f9-9f9e-4bb1-9eb1-fd1b0241989f","Type":"ContainerStarted","Data":"bd5e642a85813d20414b86bad37d65fe620e2d8be41ad21a3fb1502b8dbb8626"}
Sep 17 21:58:46.352928 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:46.352901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs" event={"ID":"425aa0ec-46a3-4712-9a04-91b742dde2b6","Type":"ContainerStarted","Data":"12a0f27613eb2e1404d4eea1d9eaf2ed2d477193c95138a87df7f35ad6c6bb1c"}
Sep 17 21:58:46.373622 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:46.373572    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wxxs" podStartSLOduration=3.907901026 podStartE2EDuration="23.373552283s" totalImagesPullingTime="19.465651257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:46.372886427 +0000 UTC m=+25.456474396" watchObservedRunningTime="2026-09-17 21:58:46.373552283 +0000 UTC m=+25.457140231"
Sep 17 21:58:47.144862 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:47.144818    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:47.145531 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:47.145508    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:47.180810 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:47.180786    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:47.180960 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:47.180789    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:47.180960 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:47.180900    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:47.181087 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:47.180785    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:47.181087 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:47.180970    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:47.181208 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:47.181083    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:49.180843 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:49.180653    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:49.181524 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:49.180722    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:49.181524 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:49.180943    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:49.181524 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:49.180653    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:49.181524 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:49.180985    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:49.181524 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:49.181064    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:49.362244 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:49.362221    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 21:58:49.362627 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:49.362596    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct" event={"ID":"628981f9-9f9e-4bb1-9eb1-fd1b0241989f","Type":"ContainerStarted","Data":"de635a97bff81ca30d232ed80802e04a57f1b1d4d19778be8b2c079d6cdb0034"}
Sep 17 21:58:49.362923 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:49.362902    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:49.363019 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:49.362972    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:49.363078 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:49.363060    2583 scope.go:117] "RemoveContainer" containerID="24476b0227789387b38ca0a969171b10e6896b28046396f6f8b08ea9abe021b1"
Sep 17 21:58:49.364634 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:49.364611    2583 generic.go:356] "Generic (PLEG): container finished" podID="6df8b67b-5e30-4cbb-8495-6f78f2524e13" containerID="40ac57026a1254440712e59a4b9b3aee177799c91080bcc922f404a96c0a77b6" exitCode=0
Sep 17 21:58:49.364716 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:49.364642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlb9z" event={"ID":"6df8b67b-5e30-4cbb-8495-6f78f2524e13","Type":"ContainerDied","Data":"40ac57026a1254440712e59a4b9b3aee177799c91080bcc922f404a96c0a77b6"}
Sep 17 21:58:49.379834 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:49.379818    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:50.371039 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:50.371017    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 21:58:50.371381 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:50.371359    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct" event={"ID":"628981f9-9f9e-4bb1-9eb1-fd1b0241989f","Type":"ContainerStarted","Data":"faa2b6cf2461b84cc6ef2f177d45bae800a544fd46e17d4f21a7861f19eb4e34"}
Sep 17 21:58:50.371562 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:50.371546    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:50.390377 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:50.390354    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:58:50.403054 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:50.403014    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct" podStartSLOduration=9.997076482 podStartE2EDuration="27.402999956s" totalImagesPullingTime="17.405923474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:58:50.401654843 +0000 UTC m=+29.485242811" watchObservedRunningTime="2026-09-17 21:58:50.402999956 +0000 UTC m=+29.486587904"
Sep 17 21:58:50.653858 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:50.653692    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8xvj"]
Sep 17 21:58:50.654073 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:50.653955    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:50.654073 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:50.654056    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:50.657231 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:50.657204    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rffj6"]
Sep 17 21:58:50.657346 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:50.657308    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:50.657454 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:50.657432    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:50.657884 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:50.657867    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n64gh"]
Sep 17 21:58:50.657961 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:50.657949    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:50.658047 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:50.658027    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:51.375299 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:51.375265    2583 generic.go:356] "Generic (PLEG): container finished" podID="6df8b67b-5e30-4cbb-8495-6f78f2524e13" containerID="2ed627d958e8eb8dd814a7c9eb2ab8315700ae89ca1f6d67768cc5944d60498a" exitCode=0
Sep 17 21:58:51.375661 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:51.375341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlb9z" event={"ID":"6df8b67b-5e30-4cbb-8495-6f78f2524e13","Type":"ContainerDied","Data":"2ed627d958e8eb8dd814a7c9eb2ab8315700ae89ca1f6d67768cc5944d60498a"}
Sep 17 21:58:52.180591 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:52.180560    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:52.180732 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:52.180567    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:52.180732 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:52.180651    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:52.180732 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:52.180567    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:52.180834 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:52.180735    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:52.180834 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:52.180804    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:52.820989 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:52.820906    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:52.925372 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:52.925344    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:53.033775 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:53.033740    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:53.134309 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:53.134279    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:53.382344 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:53.382311    2583 generic.go:356] "Generic (PLEG): container finished" podID="6df8b67b-5e30-4cbb-8495-6f78f2524e13" containerID="008a6c5e70cbd9231966baf575267c215d9acfdbe93fcb03573e9c9afa58f89b" exitCode=0
Sep 17 21:58:53.382510 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:53.382357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlb9z" event={"ID":"6df8b67b-5e30-4cbb-8495-6f78f2524e13","Type":"ContainerDied","Data":"008a6c5e70cbd9231966baf575267c215d9acfdbe93fcb03573e9c9afa58f89b"}
Sep 17 21:58:53.574416 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:53.574381    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:53.642921 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:53.642851    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:54.133998 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:54.133962    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:54.135686 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:54.134159    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 21:58:54.135686 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:54.134892    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lf995"
Sep 17 21:58:54.180243 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:54.180207    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:54.180243 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:54.180227    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:54.180243 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:54.180240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:54.180576 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:54.180330    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:54.180576 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:54.180410    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:54.180576 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:54.180481    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:54.241658 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:54.241619    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:56.180956 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.180886    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:56.180956 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.180906    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:56.180956 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.180885    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:56.181562 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.181034    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n64gh" podUID="6cc2ec45-0451-46da-aa90-214fd10d39bd"
Sep 17 21:58:56.181562 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.181099    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8xvj" podUID="032d58e2-2e31-40c8-87cc-a4d5939db410"
Sep 17 21:58:56.181562 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.181124    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rffj6" podUID="7f93f48f-8169-4a8b-885c-b591f433f601"
Sep 17 21:58:56.346823 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.346788    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeReady"
Sep 17 21:58:56.347006 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.346951    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 21:58:56.398039 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.398007    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nvdzb"]
Sep 17 21:58:56.417281 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.417258    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2x2z9"]
Sep 17 21:58:56.417448 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.417392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nvdzb"
Sep 17 21:58:56.419935 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.419914    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 21:58:56.420398 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.420378    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 21:58:56.420507 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.420490    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 21:58:56.420666 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.420644    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hxw4c\""
Sep 17 21:58:56.427718 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.427700    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nvdzb"]
Sep 17 21:58:56.427822 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.427725    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2x2z9"]
Sep 17 21:58:56.427822 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.427799    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.430311 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.430295    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cb92b\""
Sep 17 21:58:56.430452 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.430434    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 21:58:56.430759 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.430743    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 21:58:56.510192 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.510128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqg9f\" (UniqueName: \"kubernetes.io/projected/370fad22-3b0a-489b-a3f9-f91ff54de074-kube-api-access-xqg9f\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.510361 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.510237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/370fad22-3b0a-489b-a3f9-f91ff54de074-config-volume\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.510361 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.510259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.510361 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.510276    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kr7\" (UniqueName: \"kubernetes.io/projected/d98da831-8421-4244-819f-08a1f4a46839-kube-api-access-b6kr7\") pod \"ingress-canary-nvdzb\" (UID: \"d98da831-8421-4244-819f-08a1f4a46839\") " pod="openshift-ingress-canary/ingress-canary-nvdzb"
Sep 17 21:58:56.510361 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.510296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d98da831-8421-4244-819f-08a1f4a46839-cert\") pod \"ingress-canary-nvdzb\" (UID: \"d98da831-8421-4244-819f-08a1f4a46839\") " pod="openshift-ingress-canary/ingress-canary-nvdzb"
Sep 17 21:58:56.510361 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.510311    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370fad22-3b0a-489b-a3f9-f91ff54de074-tmp-dir\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.611501 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.611459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/370fad22-3b0a-489b-a3f9-f91ff54de074-config-volume\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.611501 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.611507    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.611786 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.611609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kr7\" (UniqueName: \"kubernetes.io/projected/d98da831-8421-4244-819f-08a1f4a46839-kube-api-access-b6kr7\") pod \"ingress-canary-nvdzb\" (UID: \"d98da831-8421-4244-819f-08a1f4a46839\") " pod="openshift-ingress-canary/ingress-canary-nvdzb"
Sep 17 21:58:56.611786 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.611620    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:56.611786 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.611749    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls podName:370fad22-3b0a-489b-a3f9-f91ff54de074 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:57.111720198 +0000 UTC m=+36.195308141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls") pod "dns-default-2x2z9" (UID: "370fad22-3b0a-489b-a3f9-f91ff54de074") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:56.611786 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.611655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d98da831-8421-4244-819f-08a1f4a46839-cert\") pod \"ingress-canary-nvdzb\" (UID: \"d98da831-8421-4244-819f-08a1f4a46839\") " pod="openshift-ingress-canary/ingress-canary-nvdzb"
Sep 17 21:58:56.611981 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.611802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370fad22-3b0a-489b-a3f9-f91ff54de074-tmp-dir\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.611981 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.611877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqg9f\" (UniqueName: \"kubernetes.io/projected/370fad22-3b0a-489b-a3f9-f91ff54de074-kube-api-access-xqg9f\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.612130 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.612107    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370fad22-3b0a-489b-a3f9-f91ff54de074-tmp-dir\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.612206 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.612126    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/370fad22-3b0a-489b-a3f9-f91ff54de074-config-volume\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.616284 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.616109    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d98da831-8421-4244-819f-08a1f4a46839-cert\") pod \"ingress-canary-nvdzb\" (UID: \"d98da831-8421-4244-819f-08a1f4a46839\") " pod="openshift-ingress-canary/ingress-canary-nvdzb"
Sep 17 21:58:56.620837 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.620819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqg9f\" (UniqueName: \"kubernetes.io/projected/370fad22-3b0a-489b-a3f9-f91ff54de074-kube-api-access-xqg9f\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:56.621029 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.621007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kr7\" (UniqueName: \"kubernetes.io/projected/d98da831-8421-4244-819f-08a1f4a46839-kube-api-access-b6kr7\") pod \"ingress-canary-nvdzb\" (UID: \"d98da831-8421-4244-819f-08a1f4a46839\") " pod="openshift-ingress-canary/ingress-canary-nvdzb"
Sep 17 21:58:56.727860 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.727790    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nvdzb"
Sep 17 21:58:56.812263 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.812213    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:56.813170 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.813150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:56.813252 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.813228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:56.813309 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.813283    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:56.813362 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.813320    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:56.813362 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.813350    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret podName:7f93f48f-8169-4a8b-885c-b591f433f601 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:28.813333551 +0000 UTC m=+67.896921495 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret") pod "global-pull-secret-syncer-rffj6" (UID: "7f93f48f-8169-4a8b-885c-b591f433f601") : object "kube-system"/"original-pull-secret" not registered
Sep 17 21:58:56.813478 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.813369    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs podName:6cc2ec45-0451-46da-aa90-214fd10d39bd nodeName:}" failed. No retries permitted until 2026-09-17 21:59:28.813360121 +0000 UTC m=+67.896948048 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs") pod "network-metrics-daemon-n64gh" (UID: "6cc2ec45-0451-46da-aa90-214fd10d39bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 21:58:56.908234 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.908207    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:56.914286 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:56.914251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784fz\" (UniqueName: \"kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz\") pod \"network-check-target-q8xvj\" (UID: \"032d58e2-2e31-40c8-87cc-a4d5939db410\") " pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:56.914398 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.914385    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 21:58:56.914453 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.914403    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 21:58:56.914453 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.914413    2583 projected.go:196] Error preparing data for projected volume kube-api-access-784fz for pod openshift-network-diagnostics/network-check-target-q8xvj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:56.914547 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:56.914469    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz podName:032d58e2-2e31-40c8-87cc-a4d5939db410 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:28.914450521 +0000 UTC m=+67.998038465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-784fz" (UniqueName: "kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz") pod "network-check-target-q8xvj" (UID: "032d58e2-2e31-40c8-87cc-a4d5939db410") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 21:58:57.115947 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:57.115845    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:57.116195 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:57.116015    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:57.116195 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:57.116091    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls podName:370fad22-3b0a-489b-a3f9-f91ff54de074 nodeName:}" failed. No retries permitted until 2026-09-17 21:58:58.116074116 +0000 UTC m=+37.199662047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls") pod "dns-default-2x2z9" (UID: "370fad22-3b0a-489b-a3f9-f91ff54de074") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:57.646280 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:57.646251    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nvdzb"]
Sep 17 21:58:57.649540 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:58:57.649500    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98da831_8421_4244_819f_08a1f4a46839.slice/crio-fb7d13ab1ce91eee85ec0b923cae8b46c696ab6494dd0d2dd868963995fc857c WatchSource:0}: Error finding container fb7d13ab1ce91eee85ec0b923cae8b46c696ab6494dd0d2dd868963995fc857c: Status 404 returned error can't find the container with id fb7d13ab1ce91eee85ec0b923cae8b46c696ab6494dd0d2dd868963995fc857c
Sep 17 21:58:58.124848 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.124764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:58:58.124982 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:58.124886    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:58:58.124982 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:58:58.124938    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls podName:370fad22-3b0a-489b-a3f9-f91ff54de074 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:00.124924549 +0000 UTC m=+39.208512476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls") pod "dns-default-2x2z9" (UID: "370fad22-3b0a-489b-a3f9-f91ff54de074") : secret "dns-default-metrics-tls" not found
Sep 17 21:58:58.180827 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.180794    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:58:58.180985 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.180794    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:58:58.180985 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.180794    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:58:58.183718 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.183698    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qd56j\""
Sep 17 21:58:58.184458 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.184440    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:58:58.184458 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.184453    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:58:58.184588 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.184480    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:58:58.184727 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.184712    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pzq2x\""
Sep 17 21:58:58.184767 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.184736    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:58:58.398651 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.398614    2583 generic.go:356] "Generic (PLEG): container finished" podID="6df8b67b-5e30-4cbb-8495-6f78f2524e13" containerID="142091992f1fa58b6576c9312d84ab08dade8da8181fafdbd148a64592e0676f" exitCode=0
Sep 17 21:58:58.398827 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.398696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlb9z" event={"ID":"6df8b67b-5e30-4cbb-8495-6f78f2524e13","Type":"ContainerDied","Data":"142091992f1fa58b6576c9312d84ab08dade8da8181fafdbd148a64592e0676f"}
Sep 17 21:58:58.400044 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:58.399863    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nvdzb" event={"ID":"d98da831-8421-4244-819f-08a1f4a46839","Type":"ContainerStarted","Data":"fb7d13ab1ce91eee85ec0b923cae8b46c696ab6494dd0d2dd868963995fc857c"}
Sep 17 21:58:59.387942 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:59.386876    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:58:59.405840 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:59.405809    2583 generic.go:356] "Generic (PLEG): container finished" podID="6df8b67b-5e30-4cbb-8495-6f78f2524e13" containerID="40234a507abd68fdfa0b1fd9efaa2b604567d076e965f7824560f28f272fd494" exitCode=0
Sep 17 21:58:59.405978 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:59.405849    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlb9z" event={"ID":"6df8b67b-5e30-4cbb-8495-6f78f2524e13","Type":"ContainerDied","Data":"40234a507abd68fdfa0b1fd9efaa2b604567d076e965f7824560f28f272fd494"}
Sep 17 21:58:59.446363 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:58:59.446332    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:00.142235 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:00.142008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:59:00.142380 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:00.142167    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:00.142380 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:00.142352    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls podName:370fad22-3b0a-489b-a3f9-f91ff54de074 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:04.142337354 +0000 UTC m=+43.225925291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls") pod "dns-default-2x2z9" (UID: "370fad22-3b0a-489b-a3f9-f91ff54de074") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:00.411460 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:00.411379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlb9z" event={"ID":"6df8b67b-5e30-4cbb-8495-6f78f2524e13","Type":"ContainerStarted","Data":"ca18d06da9903cb0ee050cfef1fd86cf8d40158cd2fe20e6ab0d9918fd087bce"}
Sep 17 21:59:00.412820 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:00.412795    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nvdzb" event={"ID":"d98da831-8421-4244-819f-08a1f4a46839","Type":"ContainerStarted","Data":"1492d92b65d6d444bfa2e37e9b1c09847c1bf43bd5b51a292b6383d4ed40dd89"}
Sep 17 21:59:00.436492 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:00.436448    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wlb9z" podStartSLOduration=9.713692801 podStartE2EDuration="37.436434029s" totalImagesPullingTime="27.722741228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:00.434918557 +0000 UTC m=+39.518506541" watchObservedRunningTime="2026-09-17 21:59:00.436434029 +0000 UTC m=+39.520021978"
Sep 17 21:59:00.453969 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:00.453928    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nvdzb" podStartSLOduration=2.306537398 podStartE2EDuration="4.453916391s" totalImagesPullingTime="2.147378993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:00.453230346 +0000 UTC m=+39.536818295" watchObservedRunningTime="2026-09-17 21:59:00.453916391 +0000 UTC m=+39.537504340"
Sep 17 21:59:00.546330 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:00.546303    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:00.600833 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:00.600805    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:01.107177 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:01.107131    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:01.171005 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:01.170976    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:04.173183 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:04.173144    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:59:04.173552 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:04.173275    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:04.173552 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:04.173325    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls podName:370fad22-3b0a-489b-a3f9-f91ff54de074 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:12.173312513 +0000 UTC m=+51.256900440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls") pod "dns-default-2x2z9" (UID: "370fad22-3b0a-489b-a3f9-f91ff54de074") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:12.228157 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:12.228096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:59:12.228639 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:12.228278    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:12.228639 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:12.228364    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls podName:370fad22-3b0a-489b-a3f9-f91ff54de074 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:28.228342018 +0000 UTC m=+67.311929966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls") pod "dns-default-2x2z9" (UID: "370fad22-3b0a-489b-a3f9-f91ff54de074") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:13.083520 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:13.083483    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:16.010945 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.010912    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:16.070064 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.070041    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:16.227337 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.227310    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-mhdkx"]
Sep 17 21:59:16.264281 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.264211    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-mhdkx"]
Sep 17 21:59:16.264403 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.264298    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.267119 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.267095    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 21:59:16.267119 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.267107    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 21:59:16.267314 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.267121    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 21:59:16.267314 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.267186    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 21:59:16.267386 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.267319    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qnf6v\""
Sep 17 21:59:16.272404 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.272387    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 21:59:16.331913 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.331885    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-2kxrz"]
Sep 17 21:59:16.352373 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.352347    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-2kxrz"]
Sep 17 21:59:16.352489 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.352436    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.355466 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.355444    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 21:59:16.356024 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356001    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 21:59:16.356129 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356063    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 21:59:16.356129 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356063    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 21:59:16.356343 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrjq9\" (UniqueName: \"kubernetes.io/projected/0a9a9e08-09a2-4062-9b80-d293592d8df2-kube-api-access-xrjq9\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.356421 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356341    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-sd74w\""
Sep 17 21:59:16.356421 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356346    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmf4r\" (UniqueName: \"kubernetes.io/projected/a5201b1b-c03b-47fc-806d-cce55f2db728-kube-api-access-kmf4r\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.356421 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356386    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a5201b1b-c03b-47fc-806d-cce55f2db728-serving-cert\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.356421 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5201b1b-c03b-47fc-806d-cce55f2db728-trusted-ca\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.356593 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9e08-09a2-4062-9b80-d293592d8df2-tmp\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.356593 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a9a9e08-09a2-4062-9b80-d293592d8df2-serving-cert\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.356593 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356504    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9a9e08-09a2-4062-9b80-d293592d8df2-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.356593 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9e08-09a2-4062-9b80-d293592d8df2-snapshots\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.356593 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356551    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9a9e08-09a2-4062-9b80-d293592d8df2-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.356759 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.356594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5201b1b-c03b-47fc-806d-cce55f2db728-config\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.378549 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.378523    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 21:59:16.430931 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.430907    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c8d9d759d-mkskv"]
Sep 17 21:59:16.444052 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.444025    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.446535 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.446517    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 21:59:16.446821 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.446777    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bsxrc\""
Sep 17 21:59:16.446821 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.446791    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 21:59:16.446821 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.446799    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 21:59:16.446821 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.446815    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 21:59:16.447016 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.446815    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 21:59:16.447016 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.446791    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 21:59:16.451621 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.451599    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c8d9d759d-mkskv"]
Sep 17 21:59:16.456918 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.456899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9e08-09a2-4062-9b80-d293592d8df2-snapshots\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.457056 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.456935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9a9e08-09a2-4062-9b80-d293592d8df2-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.457056 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.456950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5201b1b-c03b-47fc-806d-cce55f2db728-config\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.457056 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.456973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-stats-auth\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.457056 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.456988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngb7w\" (UniqueName: \"kubernetes.io/projected/b7875b9c-d8af-4a71-97ff-b5489c981863-kube-api-access-ngb7w\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.457056 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrjq9\" (UniqueName: \"kubernetes.io/projected/0a9a9e08-09a2-4062-9b80-d293592d8df2-kube-api-access-xrjq9\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.457333 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmf4r\" (UniqueName: \"kubernetes.io/projected/a5201b1b-c03b-47fc-806d-cce55f2db728-kube-api-access-kmf4r\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.457333 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-default-certificate\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.457333 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a5201b1b-c03b-47fc-806d-cce55f2db728-serving-cert\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.457333 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.457333 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5201b1b-c03b-47fc-806d-cce55f2db728-trusted-ca\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.457333 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457317    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.457603 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9e08-09a2-4062-9b80-d293592d8df2-tmp\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.457603 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a9a9e08-09a2-4062-9b80-d293592d8df2-serving-cert\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.457603 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9a9e08-09a2-4062-9b80-d293592d8df2-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.457603 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9e08-09a2-4062-9b80-d293592d8df2-snapshots\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.457807 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a5201b1b-c03b-47fc-806d-cce55f2db728-config\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.457807 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a9a9e08-09a2-4062-9b80-d293592d8df2-tmp\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.457954 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9a9e08-09a2-4062-9b80-d293592d8df2-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.458013 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.457984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a9a9e08-09a2-4062-9b80-d293592d8df2-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.458151 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.458117    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5201b1b-c03b-47fc-806d-cce55f2db728-trusted-ca\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.460959 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.460939    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a9a9e08-09a2-4062-9b80-d293592d8df2-serving-cert\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.461048 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.461012    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a5201b1b-c03b-47fc-806d-cce55f2db728-serving-cert\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.469940 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.469915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmf4r\" (UniqueName: \"kubernetes.io/projected/a5201b1b-c03b-47fc-806d-cce55f2db728-kube-api-access-kmf4r\") pod \"console-operator-fd4bb46c-2kxrz\" (UID: \"a5201b1b-c03b-47fc-806d-cce55f2db728\") " pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.470150 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.470111    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrjq9\" (UniqueName: \"kubernetes.io/projected/0a9a9e08-09a2-4062-9b80-d293592d8df2-kube-api-access-xrjq9\") pod \"insights-operator-6f9c644dd9-mhdkx\" (UID: \"0a9a9e08-09a2-4062-9b80-d293592d8df2\") " pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.558435 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.558355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-stats-auth\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.558435 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.558388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngb7w\" (UniqueName: \"kubernetes.io/projected/b7875b9c-d8af-4a71-97ff-b5489c981863-kube-api-access-ngb7w\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.558435 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.558411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-default-certificate\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.558435 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.558436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.558733 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:16.558508    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 21:59:16.558733 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.558558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.558733 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:16.558564    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs podName:b7875b9c-d8af-4a71-97ff-b5489c981863 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:17.058544308 +0000 UTC m=+56.142132236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs") pod "router-default-5c8d9d759d-mkskv" (UID: "b7875b9c-d8af-4a71-97ff-b5489c981863") : secret "router-metrics-certs-default" not found
Sep 17 21:59:16.558733 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:16.558720    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle podName:b7875b9c-d8af-4a71-97ff-b5489c981863 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:17.058699816 +0000 UTC m=+56.142287748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle") pod "router-default-5c8d9d759d-mkskv" (UID: "b7875b9c-d8af-4a71-97ff-b5489c981863") : configmap references non-existent config key: service-ca.crt
Sep 17 21:59:16.559529 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.559506    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:16.560777 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.560757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-default-certificate\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.560917 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.560897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-stats-auth\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.568520 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.568496    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngb7w\" (UniqueName: \"kubernetes.io/projected/b7875b9c-d8af-4a71-97ff-b5489c981863-kube-api-access-ngb7w\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:16.574413 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.574397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx"
Sep 17 21:59:16.655582 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.655550    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:16.663367 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.663344    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:16.877050 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.876977    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-mhdkx"]
Sep 17 21:59:16.880028 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:16.880001    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9a9e08_09a2_4062_9b80_d293592d8df2.slice/crio-d0b2f2eb17d0d928fd64527653613f1c33e5a1a9d91e11f605d170da597c2f46 WatchSource:0}: Error finding container d0b2f2eb17d0d928fd64527653613f1c33e5a1a9d91e11f605d170da597c2f46: Status 404 returned error can't find the container with id d0b2f2eb17d0d928fd64527653613f1c33e5a1a9d91e11f605d170da597c2f46
Sep 17 21:59:16.943088 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:16.943059    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-2kxrz"]
Sep 17 21:59:16.945776 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:16.945749    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5201b1b_c03b_47fc_806d_cce55f2db728.slice/crio-472c53b16d13155ae7a1aa2b5bc5ddff57029f7ca874b0d04076771d59c9b4bb WatchSource:0}: Error finding container 472c53b16d13155ae7a1aa2b5bc5ddff57029f7ca874b0d04076771d59c9b4bb: Status 404 returned error can't find the container with id 472c53b16d13155ae7a1aa2b5bc5ddff57029f7ca874b0d04076771d59c9b4bb
Sep 17 21:59:17.062114 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:17.062079    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:17.062114 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:17.062119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:17.062542 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:17.062242    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 21:59:17.062542 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:17.062313    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle podName:b7875b9c-d8af-4a71-97ff-b5489c981863 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:18.062296241 +0000 UTC m=+57.145884168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle") pod "router-default-5c8d9d759d-mkskv" (UID: "b7875b9c-d8af-4a71-97ff-b5489c981863") : configmap references non-existent config key: service-ca.crt
Sep 17 21:59:17.062542 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:17.062337    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs podName:b7875b9c-d8af-4a71-97ff-b5489c981863 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:18.062329018 +0000 UTC m=+57.145916944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs") pod "router-default-5c8d9d759d-mkskv" (UID: "b7875b9c-d8af-4a71-97ff-b5489c981863") : secret "router-metrics-certs-default" not found
Sep 17 21:59:17.454576 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:17.454540    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx" event={"ID":"0a9a9e08-09a2-4062-9b80-d293592d8df2","Type":"ContainerStarted","Data":"d0b2f2eb17d0d928fd64527653613f1c33e5a1a9d91e11f605d170da597c2f46"}
Sep 17 21:59:17.456362 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:17.456326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz" event={"ID":"a5201b1b-c03b-47fc-806d-cce55f2db728","Type":"ContainerStarted","Data":"472c53b16d13155ae7a1aa2b5bc5ddff57029f7ca874b0d04076771d59c9b4bb"}
Sep 17 21:59:18.071041 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:18.071002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:18.071462 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:18.071056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:18.071462 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:18.071166    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 21:59:18.071462 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:18.071199    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle podName:b7875b9c-d8af-4a71-97ff-b5489c981863 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:20.071178911 +0000 UTC m=+59.154766843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle") pod "router-default-5c8d9d759d-mkskv" (UID: "b7875b9c-d8af-4a71-97ff-b5489c981863") : configmap references non-existent config key: service-ca.crt
Sep 17 21:59:18.071462 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:18.071221    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs podName:b7875b9c-d8af-4a71-97ff-b5489c981863 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:20.071212498 +0000 UTC m=+59.154800439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs") pod "router-default-5c8d9d759d-mkskv" (UID: "b7875b9c-d8af-4a71-97ff-b5489c981863") : secret "router-metrics-certs-default" not found
Sep 17 21:59:18.839902 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:18.839874    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:18.903981 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:18.903941    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:20.090040 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.089948    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:20.090040 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.090000    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:20.090467 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:20.090102    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 21:59:20.090467 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:20.090187    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs podName:b7875b9c-d8af-4a71-97ff-b5489c981863 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:24.090169111 +0000 UTC m=+63.173757048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs") pod "router-default-5c8d9d759d-mkskv" (UID: "b7875b9c-d8af-4a71-97ff-b5489c981863") : secret "router-metrics-certs-default" not found
Sep 17 21:59:20.090467 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:20.090216    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle podName:b7875b9c-d8af-4a71-97ff-b5489c981863 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:24.090198802 +0000 UTC m=+63.173786734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle") pod "router-default-5c8d9d759d-mkskv" (UID: "b7875b9c-d8af-4a71-97ff-b5489c981863") : configmap references non-existent config key: service-ca.crt
Sep 17 21:59:20.465828 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.465799    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/0.log"
Sep 17 21:59:20.465994 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.465836    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5201b1b-c03b-47fc-806d-cce55f2db728" containerID="a3ee576848d35582efb32a283046263f1a705b726ee605c531b65b4fc3eb46e6" exitCode=255
Sep 17 21:59:20.465994 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.465867    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz" event={"ID":"a5201b1b-c03b-47fc-806d-cce55f2db728","Type":"ContainerDied","Data":"a3ee576848d35582efb32a283046263f1a705b726ee605c531b65b4fc3eb46e6"}
Sep 17 21:59:20.466171 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.466155    2583 scope.go:117] "RemoveContainer" containerID="a3ee576848d35582efb32a283046263f1a705b726ee605c531b65b4fc3eb46e6"
Sep 17 21:59:20.467571 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.467547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx" event={"ID":"0a9a9e08-09a2-4062-9b80-d293592d8df2","Type":"ContainerStarted","Data":"1fd235404a8136fb233a8f95244208719c34509301b14d7bcf45491f723ff255"}
Sep 17 21:59:20.498003 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.497958    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx" podStartSLOduration=1.682020617 podStartE2EDuration="4.497944706s" totalImagesPullingTime="2.815924089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:20.497324666 +0000 UTC m=+59.580912615" watchObservedRunningTime="2026-09-17 21:59:20.497944706 +0000 UTC m=+59.581532655"
Sep 17 21:59:20.817235 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.817123    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:20.854277 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.854248    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zdc4m"]
Sep 17 21:59:20.876624 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.876600    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.879471 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.879443    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdc4m"]
Sep 17 21:59:20.897632 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.897603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.897750 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.897641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-data-volume\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.897750 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.897715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-crio-socket\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.897750 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.897744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.897878 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.897797    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2js\" (UniqueName: \"kubernetes.io/projected/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-api-access-dr2js\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.898678 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.898660    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 21:59:20.898745 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.898732    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wm66n\""
Sep 17 21:59:20.907218 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.907202    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 21:59:20.930613 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.930582    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:20.998365 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.998344    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2js\" (UniqueName: \"kubernetes.io/projected/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-api-access-dr2js\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.998476 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.998391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.998476 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.998444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-data-volume\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.998476 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:20.998449    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:20.998574 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:20.998501    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm podName:69af59c2-2c6f-46c1-b9d3-ec41a11c14ee nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.498488595 +0000 UTC m=+60.582076522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdc4m" (UID: "69af59c2-2c6f-46c1-b9d3-ec41a11c14ee") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:20.998574 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.998513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-crio-socket\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.998574 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.998528    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.998691 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:20.998624    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:20.998691 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.998637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-crio-socket\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:20.998691 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:20.998669    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls podName:69af59c2-2c6f-46c1-b9d3-ec41a11c14ee nodeName:}" failed. No retries permitted until 2026-09-17 21:59:21.498656019 +0000 UTC m=+60.582243948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdc4m" (UID: "69af59c2-2c6f-46c1-b9d3-ec41a11c14ee") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:20.998810 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:20.998793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-data-volume\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:21.010570 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.010548    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2js\" (UniqueName: \"kubernetes.io/projected/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-api-access-dr2js\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:21.131855 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.131824    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:21.240147 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.240106    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:21.333221 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.333190    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:21.471800 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.471736    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 21:59:21.472147 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.472120    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/0.log"
Sep 17 21:59:21.472201 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.472166    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5201b1b-c03b-47fc-806d-cce55f2db728" containerID="b90bf0f0473557660264d1465dfdae4affa9f21881af05912775682088375eba" exitCode=255
Sep 17 21:59:21.472241 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.472193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz" event={"ID":"a5201b1b-c03b-47fc-806d-cce55f2db728","Type":"ContainerDied","Data":"b90bf0f0473557660264d1465dfdae4affa9f21881af05912775682088375eba"}
Sep 17 21:59:21.472307 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.472264    2583 scope.go:117] "RemoveContainer" containerID="a3ee576848d35582efb32a283046263f1a705b726ee605c531b65b4fc3eb46e6"
Sep 17 21:59:21.472459 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.472443    2583 scope.go:117] "RemoveContainer" containerID="b90bf0f0473557660264d1465dfdae4affa9f21881af05912775682088375eba"
Sep 17 21:59:21.472651 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:21.472630    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-2kxrz_openshift-console-operator(a5201b1b-c03b-47fc-806d-cce55f2db728)\"" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz" podUID="a5201b1b-c03b-47fc-806d-cce55f2db728"
Sep 17 21:59:21.502493 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.502475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:21.502649 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.502627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:21.502731 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:21.502712    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:21.502798 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:21.502783    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls podName:69af59c2-2c6f-46c1-b9d3-ec41a11c14ee nodeName:}" failed. No retries permitted until 2026-09-17 21:59:22.502760986 +0000 UTC m=+61.586348924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdc4m" (UID: "69af59c2-2c6f-46c1-b9d3-ec41a11c14ee") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:21.502873 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:21.502822    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:21.502930 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:21.502873    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm podName:69af59c2-2c6f-46c1-b9d3-ec41a11c14ee nodeName:}" failed. No retries permitted until 2026-09-17 21:59:22.50285677 +0000 UTC m=+61.586444718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdc4m" (UID: "69af59c2-2c6f-46c1-b9d3-ec41a11c14ee") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:21.526346 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.526328    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:21.621929 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:21.621905    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:22.190124 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:22.190092    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:22.288027 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:22.287994    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:22.390873 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:22.388200    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vpmct"
Sep 17 21:59:22.476053 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:22.475987    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 21:59:22.476307 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:22.476293    2583 scope.go:117] "RemoveContainer" containerID="b90bf0f0473557660264d1465dfdae4affa9f21881af05912775682088375eba"
Sep 17 21:59:22.476445 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:22.476428    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-2kxrz_openshift-console-operator(a5201b1b-c03b-47fc-806d-cce55f2db728)\"" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz" podUID="a5201b1b-c03b-47fc-806d-cce55f2db728"
Sep 17 21:59:22.511364 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:22.511337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:22.511455 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:22.511419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:22.511537 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:22.511518    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:22.511608 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:22.511549    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:22.511608 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:22.511572    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm podName:69af59c2-2c6f-46c1-b9d3-ec41a11c14ee nodeName:}" failed. No retries permitted until 2026-09-17 21:59:24.511557524 +0000 UTC m=+63.595145451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdc4m" (UID: "69af59c2-2c6f-46c1-b9d3-ec41a11c14ee") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:22.511608 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:22.511596    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls podName:69af59c2-2c6f-46c1-b9d3-ec41a11c14ee nodeName:}" failed. No retries permitted until 2026-09-17 21:59:24.511580117 +0000 UTC m=+63.595168061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdc4m" (UID: "69af59c2-2c6f-46c1-b9d3-ec41a11c14ee") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:22.559245 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:22.559217    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:22.628029 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:22.628007    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:22.982562 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:22.982535    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f4lz4_e9d7824d-7ae5-434d-8ccc-decace1ecd67/dns-node-resolver/0.log"
Sep 17 21:59:23.109266 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.109236    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-r9hqp"]
Sep 17 21:59:23.111954 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.111939    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp"
Sep 17 21:59:23.114407 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.114385    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 21:59:23.114519 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.114408    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9pzxn\""
Sep 17 21:59:23.115339 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.115320    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 21:59:23.115432 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.115389    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 21:59:23.115432 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.115405    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 21:59:23.123783 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.123766    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-r9hqp"]
Sep 17 21:59:23.153858 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.153811    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:23.216088 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.216064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e494b882-3f68-42ad-b2ab-974727bb0b96-signing-key\") pod \"service-ca-55f6fc7b86-r9hqp\" (UID: \"e494b882-3f68-42ad-b2ab-974727bb0b96\") " pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp"
Sep 17 21:59:23.216427 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.216158    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z9gz\" (UniqueName: \"kubernetes.io/projected/e494b882-3f68-42ad-b2ab-974727bb0b96-kube-api-access-5z9gz\") pod \"service-ca-55f6fc7b86-r9hqp\" (UID: \"e494b882-3f68-42ad-b2ab-974727bb0b96\") " pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp"
Sep 17 21:59:23.216427 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.216197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e494b882-3f68-42ad-b2ab-974727bb0b96-signing-cabundle\") pod \"service-ca-55f6fc7b86-r9hqp\" (UID: \"e494b882-3f68-42ad-b2ab-974727bb0b96\") " pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp"
Sep 17 21:59:23.222920 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.222901    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:23.317557 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.317477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e494b882-3f68-42ad-b2ab-974727bb0b96-signing-key\") pod \"service-ca-55f6fc7b86-r9hqp\" (UID: \"e494b882-3f68-42ad-b2ab-974727bb0b96\") " pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp"
Sep 17 21:59:23.317711 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.317565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z9gz\" (UniqueName: \"kubernetes.io/projected/e494b882-3f68-42ad-b2ab-974727bb0b96-kube-api-access-5z9gz\") pod \"service-ca-55f6fc7b86-r9hqp\" (UID: \"e494b882-3f68-42ad-b2ab-974727bb0b96\") " pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp"
Sep 17 21:59:23.317711 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.317608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e494b882-3f68-42ad-b2ab-974727bb0b96-signing-cabundle\") pod \"service-ca-55f6fc7b86-r9hqp\" (UID: \"e494b882-3f68-42ad-b2ab-974727bb0b96\") " pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp"
Sep 17 21:59:23.318210 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.318191    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e494b882-3f68-42ad-b2ab-974727bb0b96-signing-cabundle\") pod \"service-ca-55f6fc7b86-r9hqp\" (UID: \"e494b882-3f68-42ad-b2ab-974727bb0b96\") " pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp"
Sep 17 21:59:23.319740 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.319719    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e494b882-3f68-42ad-b2ab-974727bb0b96-signing-key\") pod \"service-ca-55f6fc7b86-r9hqp\" (UID: \"e494b882-3f68-42ad-b2ab-974727bb0b96\") " pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp"
Sep 17 21:59:23.325836 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.325812    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z9gz\" (UniqueName: \"kubernetes.io/projected/e494b882-3f68-42ad-b2ab-974727bb0b96-kube-api-access-5z9gz\") pod \"service-ca-55f6fc7b86-r9hqp\" (UID: \"e494b882-3f68-42ad-b2ab-974727bb0b96\") " pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp"
Sep 17 21:59:23.420505 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.420462    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp"
Sep 17 21:59:23.691361 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.691331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-r9hqp"]
Sep 17 21:59:23.694366 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:23.694331    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode494b882_3f68_42ad_b2ab_974727bb0b96.slice/crio-e10a71e89b508f019f190ecdd5ffa04c7378319c4688660f08cd89029c91bba2 WatchSource:0}: Error finding container e10a71e89b508f019f190ecdd5ffa04c7378319c4688660f08cd89029c91bba2: Status 404 returned error can't find the container with id e10a71e89b508f019f190ecdd5ffa04c7378319c4688660f08cd89029c91bba2
Sep 17 21:59:23.780700 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:23.780676    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qfs4z_2acdf951-0d97-4bb7-bbce-cc70bddb8e47/node-ca/0.log"
Sep 17 21:59:24.124459 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:24.124377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:24.124629 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:24.124510    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:24.124629 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:24.124562    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 21:59:24.124737 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:24.124643    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs podName:b7875b9c-d8af-4a71-97ff-b5489c981863 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:32.124621511 +0000 UTC m=+71.208209456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs") pod "router-default-5c8d9d759d-mkskv" (UID: "b7875b9c-d8af-4a71-97ff-b5489c981863") : secret "router-metrics-certs-default" not found
Sep 17 21:59:24.124737 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:24.124669    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle podName:b7875b9c-d8af-4a71-97ff-b5489c981863 nodeName:}" failed. No retries permitted until 2026-09-17 21:59:32.124652051 +0000 UTC m=+71.208239979 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle") pod "router-default-5c8d9d759d-mkskv" (UID: "b7875b9c-d8af-4a71-97ff-b5489c981863") : configmap references non-existent config key: service-ca.crt
Sep 17 21:59:24.188615 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:24.188583    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nvdzb_d98da831-8421-4244-819f-08a1f4a46839/serve-healthcheck-canary/0.log"
Sep 17 21:59:24.341639 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:24.341599    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:24.482402 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:24.482367    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp" event={"ID":"e494b882-3f68-42ad-b2ab-974727bb0b96","Type":"ContainerStarted","Data":"e10a71e89b508f019f190ecdd5ffa04c7378319c4688660f08cd89029c91bba2"}
Sep 17 21:59:24.527219 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:24.527188    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:24.527391 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:24.527239    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:24.527391 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:24.527318    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:24.527391 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:24.527360    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:24.527391 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:24.527389    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm podName:69af59c2-2c6f-46c1-b9d3-ec41a11c14ee nodeName:}" failed. No retries permitted until 2026-09-17 21:59:28.527369681 +0000 UTC m=+67.610957630 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdc4m" (UID: "69af59c2-2c6f-46c1-b9d3-ec41a11c14ee") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:24.527571 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:24.527408    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls podName:69af59c2-2c6f-46c1-b9d3-ec41a11c14ee nodeName:}" failed. No retries permitted until 2026-09-17 21:59:28.527398537 +0000 UTC m=+67.610986467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdc4m" (UID: "69af59c2-2c6f-46c1-b9d3-ec41a11c14ee") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:25.486090 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:25.486003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp" event={"ID":"e494b882-3f68-42ad-b2ab-974727bb0b96","Type":"ContainerStarted","Data":"d2a5c041712b87c63c443b91bb6298c57d08a25a3af01e3259475755ecb88010"}
Sep 17 21:59:25.506298 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:25.506247    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-r9hqp" podStartSLOduration=1.053608277 podStartE2EDuration="2.506233261s" totalImagesPullingTime="1.452624984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:25.505843404 +0000 UTC m=+64.589431374" watchObservedRunningTime="2026-09-17 21:59:25.506233261 +0000 UTC m=+64.589821209"
Sep 17 21:59:26.057946 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:26.057825    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:26.332473 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:26.332386    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:26.455195 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:26.455157    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:26.556109 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:26.556074    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:26.651057 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:26.651028    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:26.664074 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:26.664053    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:26.664205 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:26.664116    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:26.664533 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:26.664519    2583 scope.go:117] "RemoveContainer" containerID="b90bf0f0473557660264d1465dfdae4affa9f21881af05912775682088375eba"
Sep 17 21:59:26.664708 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:26.664688    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-2kxrz_openshift-console-operator(a5201b1b-c03b-47fc-806d-cce55f2db728)\"" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz" podUID="a5201b1b-c03b-47fc-806d-cce55f2db728"
Sep 17 21:59:26.757074 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:26.757046    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:27.555548 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:27.555518    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:27.656246 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:27.656214    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:28.148450 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.148416    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:28.261132 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.261091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 21:59:28.261306 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:28.261238    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 21:59:28.261306 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:28.261295    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls podName:370fad22-3b0a-489b-a3f9-f91ff54de074 nodeName:}" failed. No retries permitted until 2026-09-17 22:00:00.261280733 +0000 UTC m=+99.344868660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls") pod "dns-default-2x2z9" (UID: "370fad22-3b0a-489b-a3f9-f91ff54de074") : secret "dns-default-metrics-tls" not found
Sep 17 21:59:28.350658 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.350614    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:28.564394 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.564301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:28.564394 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.564368    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:28.564606 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:28.564420    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 21:59:28.564606 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:28.564486    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm podName:69af59c2-2c6f-46c1-b9d3-ec41a11c14ee nodeName:}" failed. No retries permitted until 2026-09-17 21:59:36.564469191 +0000 UTC m=+75.648057123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdc4m" (UID: "69af59c2-2c6f-46c1-b9d3-ec41a11c14ee") : configmap "kube-rbac-proxy" not found
Sep 17 21:59:28.564606 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:28.564506    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:28.564606 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:28.564555    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls podName:69af59c2-2c6f-46c1-b9d3-ec41a11c14ee nodeName:}" failed. No retries permitted until 2026-09-17 21:59:36.564541098 +0000 UTC m=+75.648129028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdc4m" (UID: "69af59c2-2c6f-46c1-b9d3-ec41a11c14ee") : secret "insights-runtime-extractor-tls" not found
Sep 17 21:59:28.752390 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.752359    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:28.854621 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.854532    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:28.866762 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.866739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 21:59:28.866861 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.866786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:59:28.869222 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.869205    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 21:59:28.869431 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.869414    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 21:59:28.877330 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:28.877313    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 21:59:28.877391 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:28.877368    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs podName:6cc2ec45-0451-46da-aa90-214fd10d39bd nodeName:}" failed. No retries permitted until 2026-09-17 22:00:32.877352309 +0000 UTC m=+131.960940241 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs") pod "network-metrics-daemon-n64gh" (UID: "6cc2ec45-0451-46da-aa90-214fd10d39bd") : secret "metrics-daemon-secret" not found
Sep 17 21:59:28.879386 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.879369    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7f93f48f-8169-4a8b-885c-b591f433f601-original-pull-secret\") pod \"global-pull-secret-syncer-rffj6\" (UID: \"7f93f48f-8169-4a8b-885c-b591f433f601\") " pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:59:28.953410 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.953380    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:28.967640 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.967613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784fz\" (UniqueName: \"kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz\") pod \"network-check-target-q8xvj\" (UID: \"032d58e2-2e31-40c8-87cc-a4d5939db410\") " pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:59:28.970397 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.970378    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 21:59:28.980888 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.980868    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 21:59:28.991593 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:28.991568    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-784fz\" (UniqueName: \"kubernetes.io/projected/032d58e2-2e31-40c8-87cc-a4d5939db410-kube-api-access-784fz\") pod \"network-check-target-q8xvj\" (UID: \"032d58e2-2e31-40c8-87cc-a4d5939db410\") " pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:59:29.090474 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:29.090438    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rffj6"
Sep 17 21:59:29.099009 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:29.098972    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qd56j\""
Sep 17 21:59:29.107375 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:29.107323    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:59:29.351767 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:29.351735    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:29.359782 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:29.359707    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rffj6"]
Sep 17 21:59:29.362129 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:29.362094    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f93f48f_8169_4a8b_885c_b591f433f601.slice/crio-cc6aaff0f19a426d6f7f98f18f1c51c135f75b5a33c31ad351e7564bc50a10dc WatchSource:0}: Error finding container cc6aaff0f19a426d6f7f98f18f1c51c135f75b5a33c31ad351e7564bc50a10dc: Status 404 returned error can't find the container with id cc6aaff0f19a426d6f7f98f18f1c51c135f75b5a33c31ad351e7564bc50a10dc
Sep 17 21:59:29.384834 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:29.384814    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8xvj"]
Sep 17 21:59:29.386184 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:29.386161    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032d58e2_2e31_40c8_87cc_a4d5939db410.slice/crio-5751acb7e9b66d46470192f54b791522a99e74e4d398d4d13ca0a7d5bcc805ea WatchSource:0}: Error finding container 5751acb7e9b66d46470192f54b791522a99e74e4d398d4d13ca0a7d5bcc805ea: Status 404 returned error can't find the container with id 5751acb7e9b66d46470192f54b791522a99e74e4d398d4d13ca0a7d5bcc805ea
Sep 17 21:59:29.498239 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:29.498211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rffj6" event={"ID":"7f93f48f-8169-4a8b-885c-b591f433f601","Type":"ContainerStarted","Data":"cc6aaff0f19a426d6f7f98f18f1c51c135f75b5a33c31ad351e7564bc50a10dc"}
Sep 17 21:59:29.499431 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:29.499400    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8xvj" event={"ID":"032d58e2-2e31-40c8-87cc-a4d5939db410","Type":"ContainerStarted","Data":"5751acb7e9b66d46470192f54b791522a99e74e4d398d4d13ca0a7d5bcc805ea"}
Sep 17 21:59:29.748313 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:29.748276    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:30.009005 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:30.008927    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:30.184284 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:30.184252    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:30.350368 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:30.350290    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:30.752067 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:30.752031    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:30.820505 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:30.820466    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:30.956042 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:30.956011    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:31.150666 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:31.150609    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:32.161266 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:32.161233    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:32.196703 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:32.196675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:32.196882 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:32.196737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:32.197381 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:32.197352    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7875b9c-d8af-4a71-97ff-b5489c981863-service-ca-bundle\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:32.199362 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:32.199339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7875b9c-d8af-4a71-97ff-b5489c981863-metrics-certs\") pod \"router-default-5c8d9d759d-mkskv\" (UID: \"b7875b9c-d8af-4a71-97ff-b5489c981863\") " pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:32.347318 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:32.347283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:32.355806 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:32.355780    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bsxrc\""
Sep 17 21:59:32.364086 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:32.364061    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:32.767010 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:32.766973    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:32.944861 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:32.944811    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:33.337353 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:33.337318    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:33.425766 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:33.425735    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:33.760377 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:33.760342    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:34.082177 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.078319    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:34.383186 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.383156    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:34.404851 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.404767    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c8d9d759d-mkskv"]
Sep 17 21:59:34.407312 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:34.407287    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7875b9c_d8af_4a71_97ff_b5489c981863.slice/crio-fd998ab7c254c3c3c51d025eb5826afd5ddb5fc8369cd2c4db1be1e80d4bf4db WatchSource:0}: Error finding container fd998ab7c254c3c3c51d025eb5826afd5ddb5fc8369cd2c4db1be1e80d4bf4db: Status 404 returned error can't find the container with id fd998ab7c254c3c3c51d025eb5826afd5ddb5fc8369cd2c4db1be1e80d4bf4db
Sep 17 21:59:34.516604 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.516566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8xvj" event={"ID":"032d58e2-2e31-40c8-87cc-a4d5939db410","Type":"ContainerStarted","Data":"915bec7112a54e22e607c51fd9de08a70e2553b463043302820d1f0ebac08f10"}
Sep 17 21:59:34.516768 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.516634    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 21:59:34.518112 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.518091    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c8d9d759d-mkskv" event={"ID":"b7875b9c-d8af-4a71-97ff-b5489c981863","Type":"ContainerStarted","Data":"cfd8af7f13ec3aafd4aed57495951c2e65e5a37c17a98a5936a94c351c86f016"}
Sep 17 21:59:34.518224 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.518156    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c8d9d759d-mkskv" event={"ID":"b7875b9c-d8af-4a71-97ff-b5489c981863","Type":"ContainerStarted","Data":"fd998ab7c254c3c3c51d025eb5826afd5ddb5fc8369cd2c4db1be1e80d4bf4db"}
Sep 17 21:59:34.519551 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.519526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rffj6" event={"ID":"7f93f48f-8169-4a8b-885c-b591f433f601","Type":"ContainerStarted","Data":"ad13400e0c354644e151f3e772f8293ef93121ed74e45f20e179f16b9cc63c96"}
Sep 17 21:59:34.537225 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.537184    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q8xvj" podStartSLOduration=66.79504062 podStartE2EDuration="1m11.537170326s" totalImagesPullingTime="4.742129706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:34.53623577 +0000 UTC m=+73.619823730" watchObservedRunningTime="2026-09-17 21:59:34.537170326 +0000 UTC m=+73.620758271"
Sep 17 21:59:34.557374 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.557338    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5c8d9d759d-mkskv" podStartSLOduration=18.557325915 podStartE2EDuration="18.557325915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:34.55723428 +0000 UTC m=+73.640822250" watchObservedRunningTime="2026-09-17 21:59:34.557325915 +0000 UTC m=+73.640913865"
Sep 17 21:59:34.573023 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.572987    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rffj6" podStartSLOduration=66.797153769 podStartE2EDuration="1m11.572976572s" totalImagesPullingTime="4.775822803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:34.572606129 +0000 UTC m=+73.656194093" watchObservedRunningTime="2026-09-17 21:59:34.572976572 +0000 UTC m=+73.656564521"
Sep 17 21:59:34.746394 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.746306    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:34.958043 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:34.958011    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:35.364222 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:35.364190    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:35.366734 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:35.366711    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:35.523468 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:35.523433    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:35.524622 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:35.524603    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c8d9d759d-mkskv"
Sep 17 21:59:35.551828 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:35.551806    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:35.955740 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:35.955706    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:36.150559 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:36.150525    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:36.555195 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:36.555160    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:36.630275 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:36.630187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:36.630744 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:36.630709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:36.631190 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:36.631172    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:36.632422 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:36.632405    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69af59c2-2c6f-46c1-b9d3-ec41a11c14ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdc4m\" (UID: \"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee\") " pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:36.749503 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:36.749472    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:36.788847 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:36.788816    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wm66n\""
Sep 17 21:59:36.797006 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:36.796991    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdc4m"
Sep 17 21:59:37.064379 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:37.064300    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdc4m"]
Sep 17 21:59:37.067019 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:37.066996    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69af59c2_2c6f_46c1_b9d3_ec41a11c14ee.slice/crio-ecfccc60286930e9d6f4b6ad34e359ffec685de34b9995f013422fe42226cb6d WatchSource:0}: Error finding container ecfccc60286930e9d6f4b6ad34e359ffec685de34b9995f013422fe42226cb6d: Status 404 returned error can't find the container with id ecfccc60286930e9d6f4b6ad34e359ffec685de34b9995f013422fe42226cb6d
Sep 17 21:59:37.531512 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:37.531481    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdc4m" event={"ID":"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee","Type":"ContainerStarted","Data":"f416b57ca690362b7cae84c1099c5c29eb4e70635aca6093eb61c0dc0744c17c"}
Sep 17 21:59:37.531690 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:37.531538    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdc4m" event={"ID":"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee","Type":"ContainerStarted","Data":"ecfccc60286930e9d6f4b6ad34e359ffec685de34b9995f013422fe42226cb6d"}
Sep 17 21:59:37.779900 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:37.779874    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:37.880534 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:37.880500    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:38.536412 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:38.536374    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdc4m" event={"ID":"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee","Type":"ContainerStarted","Data":"a349e25f41e898adc507725a5774435eb7e4f0fa10f9f82de176dbfa5a055074"}
Sep 17 21:59:39.543931 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:39.543895    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdc4m" event={"ID":"69af59c2-2c6f-46c1-b9d3-ec41a11c14ee","Type":"ContainerStarted","Data":"beb98c5de56462014b46316726f82ee42a98aed5257d9cfabe31985a69f223a5"}
Sep 17 21:59:39.564894 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:39.564851    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zdc4m" podStartSLOduration=17.743891923 podStartE2EDuration="19.564835904s" totalImagesPullingTime="1.820943981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:59:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:39.563957306 +0000 UTC m=+78.647545255" watchObservedRunningTime="2026-09-17 21:59:39.564835904 +0000 UTC m=+78.648423855"
Sep 17 21:59:40.180571 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:40.180527    2583 scope.go:117] "RemoveContainer" containerID="b90bf0f0473557660264d1465dfdae4affa9f21881af05912775682088375eba"
Sep 17 21:59:40.548759 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:40.548684    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 21:59:40.549095 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:40.548746    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz" event={"ID":"a5201b1b-c03b-47fc-806d-cce55f2db728","Type":"ContainerStarted","Data":"79715173eb63812edb9236ea6b625bc4ba4f95a077310a09910be4c8130c21e0"}
Sep 17 21:59:40.549341 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:40.549313    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:40.567458 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:40.567414    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz" podStartSLOduration=21.811323356 podStartE2EDuration="24.567400154s" totalImagesPullingTime="2.756076798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:40.566950298 +0000 UTC m=+79.650538266" watchObservedRunningTime="2026-09-17 21:59:40.567400154 +0000 UTC m=+79.650988102"
Sep 17 21:59:40.901994 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:40.901962    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-2kxrz"
Sep 17 21:59:41.112118 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:41.112083    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:41.212752 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:41.212676    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:41.265288 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:41.265259    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:41.594856 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:41.594773    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:41.777646 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:41.777616    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:41.843579 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:41.843551    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:42.191668 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:42.191638    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:42.251597 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:42.251564    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:42.407458 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:42.407427    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:42.468525 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:42.468452    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:42.785449 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:42.785374    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:42.885865 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:42.885837    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:42.985753 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:42.985722    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:44.399390 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:44.399359    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:44.456755 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:44.456708    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:45.891217 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.891184    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"]
Sep 17 21:59:45.893684 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.893666    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk"]
Sep 17 21:59:45.893816 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.893800    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:45.895544 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.895520    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk"
Sep 17 21:59:45.901730 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.901711    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 21:59:45.901823 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.901754    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 21:59:45.902469 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.902455    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 21:59:45.902469 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.902466    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 21:59:45.905718 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.905703    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 21:59:45.905876 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.905861    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 21:59:45.905959 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.905931    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hc4z5\""
Sep 17 21:59:45.914030 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.914011    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"]
Sep 17 21:59:45.915058 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.915035    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk"]
Sep 17 21:59:45.963066 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.963040    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cd8b94bdb-6p69t"]
Sep 17 21:59:45.964956 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.964942    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:45.967985 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.967967    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 21:59:45.968339 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.968311    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 21:59:45.968444 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.968368    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 21:59:45.968444 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.968380    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hqrds\""
Sep 17 21:59:45.974901 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.974882    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 21:59:45.985686 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:45.985663    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cd8b94bdb-6p69t"]
Sep 17 21:59:46.007609 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.007587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/875458e2-0647-414a-a8f1-eef77f709736-klusterlet-config\") pod \"klusterlet-addon-workmgr-d4fd58765-vmn2h\" (UID: \"875458e2-0647-414a-a8f1-eef77f709736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:46.007699 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.007654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/875458e2-0647-414a-a8f1-eef77f709736-tmp\") pod \"klusterlet-addon-workmgr-d4fd58765-vmn2h\" (UID: \"875458e2-0647-414a-a8f1-eef77f709736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:46.007699 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.007689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af9c964d-72a8-4bd8-9823-1eb095dc4463-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-cvfhk\" (UID: \"af9c964d-72a8-4bd8-9823-1eb095dc4463\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk"
Sep 17 21:59:46.007792 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.007716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af9c964d-72a8-4bd8-9823-1eb095dc4463-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-cvfhk\" (UID: \"af9c964d-72a8-4bd8-9823-1eb095dc4463\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk"
Sep 17 21:59:46.007827 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.007785    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4xgg\" (UniqueName: \"kubernetes.io/projected/875458e2-0647-414a-a8f1-eef77f709736-kube-api-access-l4xgg\") pod \"klusterlet-addon-workmgr-d4fd58765-vmn2h\" (UID: \"875458e2-0647-414a-a8f1-eef77f709736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:46.018498 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.018477    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cd8b94bdb-6p69t"]
Sep 17 21:59:46.018677 ip-10-0-130-119 kubenswrapper[2583]: E0917 21:59:46.018655    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-jjh5p registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-jjh5p registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t" podUID="e7f60f4d-e2d7-4134-892a-8a69599de9db"
Sep 17 21:59:46.059508 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.059483    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:46.108927 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.108900    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4xgg\" (UniqueName: \"kubernetes.io/projected/875458e2-0647-414a-a8f1-eef77f709736-kube-api-access-l4xgg\") pod \"klusterlet-addon-workmgr-d4fd58765-vmn2h\" (UID: \"875458e2-0647-414a-a8f1-eef77f709736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:46.109047 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.108951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7f60f4d-e2d7-4134-892a-8a69599de9db-ca-trust-extracted\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.109047 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.108971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-installation-pull-secrets\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.109130 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-bound-sa-token\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.109130 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-tls\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.109130 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-certificates\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.109130 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109130    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjh5p\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-kube-api-access-jjh5p\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.109290 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109174    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/875458e2-0647-414a-a8f1-eef77f709736-klusterlet-config\") pod \"klusterlet-addon-workmgr-d4fd58765-vmn2h\" (UID: \"875458e2-0647-414a-a8f1-eef77f709736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:46.109290 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/875458e2-0647-414a-a8f1-eef77f709736-tmp\") pod \"klusterlet-addon-workmgr-d4fd58765-vmn2h\" (UID: \"875458e2-0647-414a-a8f1-eef77f709736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:46.109290 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af9c964d-72a8-4bd8-9823-1eb095dc4463-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-cvfhk\" (UID: \"af9c964d-72a8-4bd8-9823-1eb095dc4463\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk"
Sep 17 21:59:46.109290 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109230    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-trusted-ca\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.109290 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af9c964d-72a8-4bd8-9823-1eb095dc4463-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-cvfhk\" (UID: \"af9c964d-72a8-4bd8-9823-1eb095dc4463\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk"
Sep 17 21:59:46.109453 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-image-registry-private-configuration\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.109739 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/875458e2-0647-414a-a8f1-eef77f709736-tmp\") pod \"klusterlet-addon-workmgr-d4fd58765-vmn2h\" (UID: \"875458e2-0647-414a-a8f1-eef77f709736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:46.109942 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.109925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af9c964d-72a8-4bd8-9823-1eb095dc4463-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-cvfhk\" (UID: \"af9c964d-72a8-4bd8-9823-1eb095dc4463\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk"
Sep 17 21:59:46.111675 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.111652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af9c964d-72a8-4bd8-9823-1eb095dc4463-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-cvfhk\" (UID: \"af9c964d-72a8-4bd8-9823-1eb095dc4463\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk"
Sep 17 21:59:46.111771 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.111703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/875458e2-0647-414a-a8f1-eef77f709736-klusterlet-config\") pod \"klusterlet-addon-workmgr-d4fd58765-vmn2h\" (UID: \"875458e2-0647-414a-a8f1-eef77f709736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:46.130440 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.130421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4xgg\" (UniqueName: \"kubernetes.io/projected/875458e2-0647-414a-a8f1-eef77f709736-kube-api-access-l4xgg\") pod \"klusterlet-addon-workmgr-d4fd58765-vmn2h\" (UID: \"875458e2-0647-414a-a8f1-eef77f709736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:46.160813 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.160762    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:46.203982 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.203958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:46.208789 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.208771    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk"
Sep 17 21:59:46.210091 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.210069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7f60f4d-e2d7-4134-892a-8a69599de9db-ca-trust-extracted\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.210180 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.210109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-installation-pull-secrets\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.210180 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.210162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-bound-sa-token\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.210305 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.210210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-tls\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.210305 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.210233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-certificates\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.210305 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.210264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjh5p\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-kube-api-access-jjh5p\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.210460 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.210309    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-trusted-ca\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.210460 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.210339    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-image-registry-private-configuration\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.211787 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.211377    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-certificates\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.211787 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.211540    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7f60f4d-e2d7-4134-892a-8a69599de9db-ca-trust-extracted\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.211787 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.211674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-trusted-ca\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.213015 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.212988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-installation-pull-secrets\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.213329 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.213258    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-image-registry-private-configuration\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.213829 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.213809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-tls\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.224050 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.224025    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-bound-sa-token\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.225166 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.225128    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjh5p\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-kube-api-access-jjh5p\") pod \"image-registry-7cd8b94bdb-6p69t\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") " pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.265258 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.265067    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:46.474106 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.474038    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"]
Sep 17 21:59:46.476959 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:46.476934    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875458e2_0647_414a_a8f1_eef77f709736.slice/crio-3a6f32425ab8e2517ce89aa8c7c54592a207b01cee706ed6fb3715d2ddbb5215 WatchSource:0}: Error finding container 3a6f32425ab8e2517ce89aa8c7c54592a207b01cee706ed6fb3715d2ddbb5215: Status 404 returned error can't find the container with id 3a6f32425ab8e2517ce89aa8c7c54592a207b01cee706ed6fb3715d2ddbb5215
Sep 17 21:59:46.504436 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.504416    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk"]
Sep 17 21:59:46.506858 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:46.506837    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9c964d_72a8_4bd8_9823_1eb095dc4463.slice/crio-65808ec258a6320eafdbc1270b18054ddc193f78767405b88848e8d3c90993f1 WatchSource:0}: Error finding container 65808ec258a6320eafdbc1270b18054ddc193f78767405b88848e8d3c90993f1: Status 404 returned error can't find the container with id 65808ec258a6320eafdbc1270b18054ddc193f78767405b88848e8d3c90993f1
Sep 17 21:59:46.567949 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.567923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h" event={"ID":"875458e2-0647-414a-a8f1-eef77f709736","Type":"ContainerStarted","Data":"3a6f32425ab8e2517ce89aa8c7c54592a207b01cee706ed6fb3715d2ddbb5215"}
Sep 17 21:59:46.568981 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.568954    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk" event={"ID":"af9c964d-72a8-4bd8-9823-1eb095dc4463","Type":"ContainerStarted","Data":"65808ec258a6320eafdbc1270b18054ddc193f78767405b88848e8d3c90993f1"}
Sep 17 21:59:46.568981 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.568968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.574886 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.574869    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:46.715318 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.715291    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-tls\") pod \"e7f60f4d-e2d7-4134-892a-8a69599de9db\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") "
Sep 17 21:59:46.715318 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.715321    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-certificates\") pod \"e7f60f4d-e2d7-4134-892a-8a69599de9db\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") "
Sep 17 21:59:46.715521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.715362    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7f60f4d-e2d7-4134-892a-8a69599de9db-ca-trust-extracted\") pod \"e7f60f4d-e2d7-4134-892a-8a69599de9db\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") "
Sep 17 21:59:46.715521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.715386    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-trusted-ca\") pod \"e7f60f4d-e2d7-4134-892a-8a69599de9db\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") "
Sep 17 21:59:46.715521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.715410    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-installation-pull-secrets\") pod \"e7f60f4d-e2d7-4134-892a-8a69599de9db\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") "
Sep 17 21:59:46.715521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.715439    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-bound-sa-token\") pod \"e7f60f4d-e2d7-4134-892a-8a69599de9db\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") "
Sep 17 21:59:46.715521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.715472    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjh5p\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-kube-api-access-jjh5p\") pod \"e7f60f4d-e2d7-4134-892a-8a69599de9db\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") "
Sep 17 21:59:46.715521 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.715504    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-image-registry-private-configuration\") pod \"e7f60f4d-e2d7-4134-892a-8a69599de9db\" (UID: \"e7f60f4d-e2d7-4134-892a-8a69599de9db\") "
Sep 17 21:59:46.717402 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.717372    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f60f4d-e2d7-4134-892a-8a69599de9db-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e7f60f4d-e2d7-4134-892a-8a69599de9db" (UID: "e7f60f4d-e2d7-4134-892a-8a69599de9db"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:59:46.717541 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.717516    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e7f60f4d-e2d7-4134-892a-8a69599de9db" (UID: "e7f60f4d-e2d7-4134-892a-8a69599de9db"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:59:46.717668 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.717609    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e7f60f4d-e2d7-4134-892a-8a69599de9db" (UID: "e7f60f4d-e2d7-4134-892a-8a69599de9db"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:59:46.719456 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.719430    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-kube-api-access-jjh5p" (OuterVolumeSpecName: "kube-api-access-jjh5p") pod "e7f60f4d-e2d7-4134-892a-8a69599de9db" (UID: "e7f60f4d-e2d7-4134-892a-8a69599de9db"). InnerVolumeSpecName "kube-api-access-jjh5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:59:46.719568 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.719468    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e7f60f4d-e2d7-4134-892a-8a69599de9db" (UID: "e7f60f4d-e2d7-4134-892a-8a69599de9db"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:59:46.719568 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.719483    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e7f60f4d-e2d7-4134-892a-8a69599de9db" (UID: "e7f60f4d-e2d7-4134-892a-8a69599de9db"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:59:46.719674 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.719658    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e7f60f4d-e2d7-4134-892a-8a69599de9db" (UID: "e7f60f4d-e2d7-4134-892a-8a69599de9db"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:59:46.719716 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.719685    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e7f60f4d-e2d7-4134-892a-8a69599de9db" (UID: "e7f60f4d-e2d7-4134-892a-8a69599de9db"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:59:46.816353 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.816294    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7f60f4d-e2d7-4134-892a-8a69599de9db-ca-trust-extracted\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 21:59:46.816353 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.816315    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-trusted-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 21:59:46.816353 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.816324    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-installation-pull-secrets\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 21:59:46.816353 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.816333    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-bound-sa-token\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 21:59:46.816353 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.816342    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjh5p\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-kube-api-access-jjh5p\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 21:59:46.816353 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.816352    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7f60f4d-e2d7-4134-892a-8a69599de9db-image-registry-private-configuration\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 21:59:46.816588 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.816360    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 21:59:46.816588 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:46.816369    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7f60f4d-e2d7-4134-892a-8a69599de9db-registry-certificates\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 21:59:47.018551 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:47.018512    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:47.573408 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:47.573357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk" event={"ID":"af9c964d-72a8-4bd8-9823-1eb095dc4463","Type":"ContainerStarted","Data":"27bcc0e5f03a009bd60c8abb5173cbf2e2ae51db69c10d45ab1f96a8b94e2b76"}
Sep 17 21:59:47.573570 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:47.573379    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cd8b94bdb-6p69t"
Sep 17 21:59:47.592747 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:47.592680    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-cvfhk" podStartSLOduration=1.653729081 podStartE2EDuration="2.592665715s" totalImagesPullingTime="938.936634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:47.592029435 +0000 UTC m=+86.675617386" watchObservedRunningTime="2026-09-17 21:59:47.592665715 +0000 UTC m=+86.676253703"
Sep 17 21:59:47.621389 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:47.621362    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cd8b94bdb-6p69t"]
Sep 17 21:59:47.625428 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:47.625406    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cd8b94bdb-6p69t"]
Sep 17 21:59:47.701258 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:47.701174    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:47.990965 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:47.990871    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:48.092377 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:48.092344    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:49.185376 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:49.185344    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7f60f4d-e2d7-4134-892a-8a69599de9db" path="/var/lib/kubelet/pods/e7f60f4d-e2d7-4134-892a-8a69599de9db/volumes"
Sep 17 21:59:50.585318 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:50.585288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h" event={"ID":"875458e2-0647-414a-a8f1-eef77f709736","Type":"ContainerStarted","Data":"797fef6cdf82eef7c8d74262bec156e503bb28f0f3aa676762543dcc726c4252"}
Sep 17 21:59:50.585684 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:50.585459    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:50.586692 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:50.586664    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h" podUID="875458e2-0647-414a-a8f1-eef77f709736" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.13:8000/readyz\": dial tcp 10.134.0.13:8000: connect: connection refused"
Sep 17 21:59:50.602082 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:50.602025    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h" podStartSLOduration=1.549885564 podStartE2EDuration="5.602006003s" totalImagesPullingTime="4.052120439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:50.600776232 +0000 UTC m=+89.684364181" watchObservedRunningTime="2026-09-17 21:59:50.602006003 +0000 UTC m=+89.685593953"
Sep 17 21:59:51.589390 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:51.589365    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4fd58765-vmn2h"
Sep 17 21:59:51.818369 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:51.818338    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:51.921928 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:51.921898    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:51.994474 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:51.994443    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf"]
Sep 17 21:59:52.000516 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.000478    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:52.000516 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.000504    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf"
Sep 17 21:59:52.003044 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.003023    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2fph5\""
Sep 17 21:59:52.003235 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.003221    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 21:59:52.007430 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.007401    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf"]
Sep 17 21:59:52.104608 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.104575    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:52.159872 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.159845    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dfd8e9ad-11d2-4bb2-88d5-d289c3d0de78-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7fxtf\" (UID: \"dfd8e9ad-11d2-4bb2-88d5-d289c3d0de78\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf"
Sep 17 21:59:52.260908 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.260829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dfd8e9ad-11d2-4bb2-88d5-d289c3d0de78-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7fxtf\" (UID: \"dfd8e9ad-11d2-4bb2-88d5-d289c3d0de78\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf"
Sep 17 21:59:52.263257 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.263228    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dfd8e9ad-11d2-4bb2-88d5-d289c3d0de78-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-7fxtf\" (UID: \"dfd8e9ad-11d2-4bb2-88d5-d289c3d0de78\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf"
Sep 17 21:59:52.309196 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.309169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf"
Sep 17 21:59:52.582024 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.581853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf"]
Sep 17 21:59:52.584467 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:52.584441    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd8e9ad_11d2_4bb2_88d5_d289c3d0de78.slice/crio-ef0336e5f52cc0b2fed8adb023402f8bc038ebb193929004f51429aea32b52bd WatchSource:0}: Error finding container ef0336e5f52cc0b2fed8adb023402f8bc038ebb193929004f51429aea32b52bd: Status 404 returned error can't find the container with id ef0336e5f52cc0b2fed8adb023402f8bc038ebb193929004f51429aea32b52bd
Sep 17 21:59:52.593023 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:52.592996    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf" event={"ID":"dfd8e9ad-11d2-4bb2-88d5-d289c3d0de78","Type":"ContainerStarted","Data":"ef0336e5f52cc0b2fed8adb023402f8bc038ebb193929004f51429aea32b52bd"}
Sep 17 21:59:54.600885 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:54.600854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf" event={"ID":"dfd8e9ad-11d2-4bb2-88d5-d289c3d0de78","Type":"ContainerStarted","Data":"6bf29089badc218a45b48d049155a81be209b29cd88bbcc4ced49db3247f70a3"}
Sep 17 21:59:54.601323 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:54.601053    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf"
Sep 17 21:59:54.605470 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:54.605442    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf"
Sep 17 21:59:54.617878 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:54.617832    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-7fxtf" podStartSLOduration=2.4637904649999998 podStartE2EDuration="3.617817151s" totalImagesPullingTime="1.154026686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:54.616454696 +0000 UTC m=+93.700042664" watchObservedRunningTime="2026-09-17 21:59:54.617817151 +0000 UTC m=+93.701405099"
Sep 17 21:59:55.022742 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.022708    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:55.055279 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.055254    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"]
Sep 17 21:59:55.057899 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.057883    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.061523 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.061499    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 21:59:55.061678 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.061518    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 21:59:55.061786 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.061561    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 21:59:55.061786 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.061569    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 21:59:55.061786 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.061573    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 21:59:55.061786 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.061578    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vcbmv\""
Sep 17 21:59:55.066349 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.066328    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"]
Sep 17 21:59:55.097313 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.097289    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:55.161344 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.161318    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:55.182470 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.182449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6674c479-6e7f-4e5b-9a68-cca01d12d656-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.182580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.182477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6674c479-6e7f-4e5b-9a68-cca01d12d656-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.182580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.182498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnbw\" (UniqueName: \"kubernetes.io/projected/6674c479-6e7f-4e5b-9a68-cca01d12d656-kube-api-access-glnbw\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.182580 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.182521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6674c479-6e7f-4e5b-9a68-cca01d12d656-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.283290 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.283220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6674c479-6e7f-4e5b-9a68-cca01d12d656-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.283290 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.283282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6674c479-6e7f-4e5b-9a68-cca01d12d656-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.283475 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.283319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnbw\" (UniqueName: \"kubernetes.io/projected/6674c479-6e7f-4e5b-9a68-cca01d12d656-kube-api-access-glnbw\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.283475 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.283347    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6674c479-6e7f-4e5b-9a68-cca01d12d656-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.283953 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.283931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6674c479-6e7f-4e5b-9a68-cca01d12d656-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.285433 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.285411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6674c479-6e7f-4e5b-9a68-cca01d12d656-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.285522 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.285494    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6674c479-6e7f-4e5b-9a68-cca01d12d656-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.292686 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.292662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnbw\" (UniqueName: \"kubernetes.io/projected/6674c479-6e7f-4e5b-9a68-cca01d12d656-kube-api-access-glnbw\") pod \"prometheus-operator-7bd5d56d6-9w6xm\" (UID: \"6674c479-6e7f-4e5b-9a68-cca01d12d656\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.367541 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.367515    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"
Sep 17 21:59:55.635332 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:55.635310    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm"]
Sep 17 21:59:55.637581 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:55.637552    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6674c479_6e7f_4e5b_9a68_cca01d12d656.slice/crio-16a2eca12a4ceefa653eef49358b2dc2608791ef164a08d90997a99379b77734 WatchSource:0}: Error finding container 16a2eca12a4ceefa653eef49358b2dc2608791ef164a08d90997a99379b77734: Status 404 returned error can't find the container with id 16a2eca12a4ceefa653eef49358b2dc2608791ef164a08d90997a99379b77734
Sep 17 21:59:56.027720 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:56.027646    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:56.086769 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:56.086737    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:56.608684 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:56.608647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm" event={"ID":"6674c479-6e7f-4e5b-9a68-cca01d12d656","Type":"ContainerStarted","Data":"16a2eca12a4ceefa653eef49358b2dc2608791ef164a08d90997a99379b77734"}
Sep 17 21:59:56.812363 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:56.812329    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:56.912832 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:56.912806    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:57.127710 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:57.127601    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:57.613897 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:57.613862    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm" event={"ID":"6674c479-6e7f-4e5b-9a68-cca01d12d656","Type":"ContainerStarted","Data":"0de5a7f16c71575ac680ad2b1c6343a8b0742bea8f4b27385d1878f4c9965165"}
Sep 17 21:59:57.614099 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:57.613912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm" event={"ID":"6674c479-6e7f-4e5b-9a68-cca01d12d656","Type":"ContainerStarted","Data":"eea9445ab1b7de6fa4fe09d38df6ef9c110eb16a6418bc7d1656c555ab7a04d2"}
Sep 17 21:59:57.659725 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:57.659668    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-9w6xm" podStartSLOduration=1.4792526480000001 podStartE2EDuration="2.65965174s" totalImagesPullingTime="1.180399092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:59:57.659356959 +0000 UTC m=+96.742944926" watchObservedRunningTime="2026-09-17 21:59:57.65965174 +0000 UTC m=+96.743239689"
Sep 17 21:59:59.165561 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.165512    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:59.389180 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.389147    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:59.413999 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.413965    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"]
Sep 17 21:59:59.420478 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.420423    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.423751 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.423724    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-84dr7\""
Sep 17 21:59:59.424262 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.424240    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 21:59:59.424556 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.424537    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 21:59:59.434295 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.433278    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zzlwg"]
Sep 17 21:59:59.439536 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.439513    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"]
Sep 17 21:59:59.439646 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.439564    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"]
Sep 17 21:59:59.439646 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.439637    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.442702 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.442513    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 21:59:59.442702 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.442553    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-krpp4\""
Sep 17 21:59:59.442702 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.442560    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 21:59:59.442702 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.442570    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 21:59:59.443860 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.443841    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.446306 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.446287    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 21:59:59.446405 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.446339    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 21:59:59.447017 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.446694    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cvtvh\""
Sep 17 21:59:59.447017 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.446814    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 21:59:59.456434 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.456412    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"]
Sep 17 21:59:59.490718 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.490675    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:59.518743 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.518719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.518848 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.518762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.518848 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.518789    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs587\" (UniqueName: \"kubernetes.io/projected/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-kube-api-access-bs587\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.518938 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.518846    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.582059 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.582034    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:59.620011 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.619992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d29f1696-1e84-4320-8218-21f95be728d2-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.620184 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-tls\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.620184 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d29f1696-1e84-4320-8218-21f95be728d2-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.620184 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d29f1696-1e84-4320-8218-21f95be728d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.620184 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.620184 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-accelerators-collector-config\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.620451 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-wtmp\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.620451 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f2123ef-58e4-400e-8708-622b61aef80b-metrics-client-ca\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.620451 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620268    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.620451 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-textfile\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.620451 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8slx8\" (UniqueName: \"kubernetes.io/projected/6f2123ef-58e4-400e-8708-622b61aef80b-kube-api-access-8slx8\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.620451 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620333    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d29f1696-1e84-4320-8218-21f95be728d2-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.620451 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d29f1696-1e84-4320-8218-21f95be728d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.620451 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f2123ef-58e4-400e-8708-622b61aef80b-sys\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.620451 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f2123ef-58e4-400e-8708-622b61aef80b-root\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.620882 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.620882 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs587\" (UniqueName: \"kubernetes.io/projected/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-kube-api-access-bs587\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.620882 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620771    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.620882 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.620806    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8qv6\" (UniqueName: \"kubernetes.io/projected/d29f1696-1e84-4320-8218-21f95be728d2-kube-api-access-b8qv6\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.621076 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.621024    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.622735 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.622718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.622818 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.622723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.629703 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.629676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs587\" (UniqueName: \"kubernetes.io/projected/1881bf5a-9323-4ce7-bb4f-31c02bb876c0-kube-api-access-bs587\") pod \"openshift-state-metrics-86c5bc79d6-hdfbf\" (UID: \"1881bf5a-9323-4ce7-bb4f-31c02bb876c0\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.722004 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.721919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d29f1696-1e84-4320-8218-21f95be728d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.722004 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.721967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f2123ef-58e4-400e-8708-622b61aef80b-sys\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722004 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.721990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f2123ef-58e4-400e-8708-622b61aef80b-root\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722222 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722059    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6f2123ef-58e4-400e-8708-622b61aef80b-root\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722222 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722064    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6f2123ef-58e4-400e-8708-622b61aef80b-sys\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722222 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722222 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8qv6\" (UniqueName: \"kubernetes.io/projected/d29f1696-1e84-4320-8218-21f95be728d2-kube-api-access-b8qv6\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.722222 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d29f1696-1e84-4320-8218-21f95be728d2-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.722436 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-tls\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722436 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d29f1696-1e84-4320-8218-21f95be728d2-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.722436 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d29f1696-1e84-4320-8218-21f95be728d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.722436 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-accelerators-collector-config\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722436 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-wtmp\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722436 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f2123ef-58e4-400e-8708-622b61aef80b-metrics-client-ca\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722436 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722422    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-textfile\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722784 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722447    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8slx8\" (UniqueName: \"kubernetes.io/projected/6f2123ef-58e4-400e-8708-622b61aef80b-kube-api-access-8slx8\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722784 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722472    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d29f1696-1e84-4320-8218-21f95be728d2-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.722784 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722630    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-wtmp\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.722784 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722696    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d29f1696-1e84-4320-8218-21f95be728d2-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.722980 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.722803    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d29f1696-1e84-4320-8218-21f95be728d2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.723226 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.723205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f2123ef-58e4-400e-8708-622b61aef80b-metrics-client-ca\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.723304 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.723283    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d29f1696-1e84-4320-8218-21f95be728d2-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.723774 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.723746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-accelerators-collector-config\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.723891 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.723873    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-textfile\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.725198 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.725122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d29f1696-1e84-4320-8218-21f95be728d2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.725198 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.725126    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-tls\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.725540 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.725518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f2123ef-58e4-400e-8708-622b61aef80b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.725734 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.725715    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d29f1696-1e84-4320-8218-21f95be728d2-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.730275 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.730254    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8slx8\" (UniqueName: \"kubernetes.io/projected/6f2123ef-58e4-400e-8708-622b61aef80b-kube-api-access-8slx8\") pod \"node-exporter-zzlwg\" (UID: \"6f2123ef-58e4-400e-8708-622b61aef80b\") " pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.730973 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.730951    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8qv6\" (UniqueName: \"kubernetes.io/projected/d29f1696-1e84-4320-8218-21f95be728d2-kube-api-access-b8qv6\") pod \"kube-state-metrics-84c885658c-j8h4q\" (UID: \"d29f1696-1e84-4320-8218-21f95be728d2\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.738732 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.738544    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"
Sep 17 21:59:59.752466 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.752283    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zzlwg"
Sep 17 21:59:59.759882 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.759850    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"
Sep 17 21:59:59.768589 ip-10-0-130-119 kubenswrapper[2583]: W0917 21:59:59.768564    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2123ef_58e4_400e_8708_622b61aef80b.slice/crio-a2f9993c79d3c19ac2c1e6df6ac75b9924933ea64416d36eff8f410aae719830 WatchSource:0}: Error finding container a2f9993c79d3c19ac2c1e6df6ac75b9924933ea64416d36eff8f410aae719830: Status 404 returned error can't find the container with id a2f9993c79d3c19ac2c1e6df6ac75b9924933ea64416d36eff8f410aae719830
Sep 17 21:59:59.776712 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.776686    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 21:59:59.958319 ip-10-0-130-119 kubenswrapper[2583]: I0917 21:59:59.958288    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.035933 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.035752    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf"]
Sep 17 22:00:00.038399 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:00:00.038359    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1881bf5a_9323_4ce7_bb4f_31c02bb876c0.slice/crio-db155da29c796c7a2b45378187da97e3d1b94552e7495742a4f8aea5b199f59d WatchSource:0}: Error finding container db155da29c796c7a2b45378187da97e3d1b94552e7495742a4f8aea5b199f59d: Status 404 returned error can't find the container with id db155da29c796c7a2b45378187da97e3d1b94552e7495742a4f8aea5b199f59d
Sep 17 22:00:00.060650 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.060627    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-j8h4q"]
Sep 17 22:00:00.062321 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.062296    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.072230 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:00:00.072204    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29f1696_1e84_4320_8218_21f95be728d2.slice/crio-a906fb83e1fed63d6fa53dc64131a8540d8d7448719d61ea23fba44b178ff1ae WatchSource:0}: Error finding container a906fb83e1fed63d6fa53dc64131a8540d8d7448719d61ea23fba44b178ff1ae: Status 404 returned error can't find the container with id a906fb83e1fed63d6fa53dc64131a8540d8d7448719d61ea23fba44b178ff1ae
Sep 17 22:00:00.162796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.162767    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.327300 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.327217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 22:00:00.330259 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.330231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370fad22-3b0a-489b-a3f9-f91ff54de074-metrics-tls\") pod \"dns-default-2x2z9\" (UID: \"370fad22-3b0a-489b-a3f9-f91ff54de074\") " pod="openshift-dns/dns-default-2x2z9"
Sep 17 22:00:00.341596 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.341576    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cb92b\""
Sep 17 22:00:00.349669 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.349641    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2x2z9"
Sep 17 22:00:00.449230 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.449184    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.549523 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.549460    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.560817 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.560789    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 22:00:00.565351 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.565317    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.568380 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.568359    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 22:00:00.570893 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.570723    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 22:00:00.578460 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.578281    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 22:00:00.578460 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.578399    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7zhvn\""
Sep 17 22:00:00.578460 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.578430    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 22:00:00.578704 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.578633    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 22:00:00.578761 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.578731    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 22:00:00.578860 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.578843    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 22:00:00.579052 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.579036    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 22:00:00.588843 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.588820    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 22:00:00.605076 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.604945    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 22:00:00.628931 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.628842    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q" event={"ID":"d29f1696-1e84-4320-8218-21f95be728d2","Type":"ContainerStarted","Data":"a906fb83e1fed63d6fa53dc64131a8540d8d7448719d61ea23fba44b178ff1ae"}
Sep 17 22:00:00.631303 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.631216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zzlwg" event={"ID":"6f2123ef-58e4-400e-8708-622b61aef80b","Type":"ContainerStarted","Data":"a2f9993c79d3c19ac2c1e6df6ac75b9924933ea64416d36eff8f410aae719830"}
Sep 17 22:00:00.634632 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.634503    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf" event={"ID":"1881bf5a-9323-4ce7-bb4f-31c02bb876c0","Type":"ContainerStarted","Data":"ac9646cbd0fea709747f375d82d0cd58e837ac572dad97362c43b3eab15a16a7"}
Sep 17 22:00:00.634632 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.634563    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf" event={"ID":"1881bf5a-9323-4ce7-bb4f-31c02bb876c0","Type":"ContainerStarted","Data":"30cf146398750685360e6b8e3810ded7435ddadd7fd4b2f6491091dd2698c568"}
Sep 17 22:00:00.634632 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.634589    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf" event={"ID":"1881bf5a-9323-4ce7-bb4f-31c02bb876c0","Type":"ContainerStarted","Data":"db155da29c796c7a2b45378187da97e3d1b94552e7495742a4f8aea5b199f59d"}
Sep 17 22:00:00.659285 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.659027    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2x2z9"]
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731533    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731563    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-web-config\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42x8x\" (UniqueName: \"kubernetes.io/projected/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-kube-api-access-42x8x\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.731753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.732516 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.731872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-config-out\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.828218 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:00:00.828176    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370fad22_3b0a_489b_a3f9_f91ff54de074.slice/crio-365a97016c966f8425f5d7dc63947b19277c9980f2927b0c29331117c4e20017 WatchSource:0}: Error finding container 365a97016c966f8425f5d7dc63947b19277c9980f2927b0c29331117c4e20017: Status 404 returned error can't find the container with id 365a97016c966f8425f5d7dc63947b19277c9980f2927b0c29331117c4e20017
Sep 17 22:00:00.832530 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.832461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.832600 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.832529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.832600 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.832558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.832691 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.832599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.832691 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.832641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-web-config\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.832856 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.832823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.832911 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.832871    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42x8x\" (UniqueName: \"kubernetes.io/projected/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-kube-api-access-42x8x\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.832967 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.832906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.832967 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.832933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.833075 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.832978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-config-out\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.833075 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.833002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.833075 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.833038    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.833075 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.833064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.833473 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.833450    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.834571 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.834544    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.835157 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.835030    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.835563 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.835538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-web-config\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.838500 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.838353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-config-out\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.838500 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.838391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.838500 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.838455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.838715 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.838502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.838715 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.838536    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.839967 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.839944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.840088 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.840067    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.840980 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.840931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.842866 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.842842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42x8x\" (UniqueName: \"kubernetes.io/projected/390aa30d-b0eb-4036-9c3b-1f8520b3e7a9-kube-api-access-42x8x\") pod \"alertmanager-main-0\" (UID: \"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.845115 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.845096    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:00.878125 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.878097    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 22:00:00.918401 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:00.918367    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:01.393428 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:01.393313    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 22:00:01.636245 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:00:01.636209    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390aa30d_b0eb_4036_9c3b_1f8520b3e7a9.slice/crio-b139ec511374d49b9b7e33cd4cbd8a4f3c98ac75a1eac0ad476175de96ba86db WatchSource:0}: Error finding container b139ec511374d49b9b7e33cd4cbd8a4f3c98ac75a1eac0ad476175de96ba86db: Status 404 returned error can't find the container with id b139ec511374d49b9b7e33cd4cbd8a4f3c98ac75a1eac0ad476175de96ba86db
Sep 17 22:00:01.640151 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:01.640108    2583 generic.go:356] "Generic (PLEG): container finished" podID="6f2123ef-58e4-400e-8708-622b61aef80b" containerID="926aca31c31eafafa32747653efa321e6e9a18309a695a8bee801e51e21fadd1" exitCode=0
Sep 17 22:00:01.640251 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:01.640168    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zzlwg" event={"ID":"6f2123ef-58e4-400e-8708-622b61aef80b","Type":"ContainerDied","Data":"926aca31c31eafafa32747653efa321e6e9a18309a695a8bee801e51e21fadd1"}
Sep 17 22:00:01.642784 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:01.642756    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf" event={"ID":"1881bf5a-9323-4ce7-bb4f-31c02bb876c0","Type":"ContainerStarted","Data":"72856b6bbf3108483bda1bb5e58e536a5a22b58275d7851980ca058e5c51ec1e"}
Sep 17 22:00:01.643881 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:01.643831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2x2z9" event={"ID":"370fad22-3b0a-489b-a3f9-f91ff54de074","Type":"ContainerStarted","Data":"365a97016c966f8425f5d7dc63947b19277c9980f2927b0c29331117c4e20017"}
Sep 17 22:00:01.688626 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:01.688583    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hdfbf" podStartSLOduration=1.699326407 podStartE2EDuration="2.688567015s" totalImagesPullingTime="989.240608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:01.687651111 +0000 UTC m=+100.771239057" watchObservedRunningTime="2026-09-17 22:00:01.688567015 +0000 UTC m=+100.772154963"
Sep 17 22:00:02.375326 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.375292    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:02.577506 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.577420    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:02.651001 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.650962    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q" event={"ID":"d29f1696-1e84-4320-8218-21f95be728d2","Type":"ContainerStarted","Data":"1afaf8624e8bced6b20200ce842b2438aa363d9657b2bf6d5c5e5f924a7e3691"}
Sep 17 22:00:02.651202 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.651040    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q" event={"ID":"d29f1696-1e84-4320-8218-21f95be728d2","Type":"ContainerStarted","Data":"cc239d78440b1a3899a2a4e28082de133ebc960c967daf1d19dc44a5956e3179"}
Sep 17 22:00:02.651202 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.651074    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q" event={"ID":"d29f1696-1e84-4320-8218-21f95be728d2","Type":"ContainerStarted","Data":"1275948b0232d58ff16f282f9fc1b1e9c7d141c56b233725a96478cfb570e3ca"}
Sep 17 22:00:02.653079 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.653051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zzlwg" event={"ID":"6f2123ef-58e4-400e-8708-622b61aef80b","Type":"ContainerStarted","Data":"e39e94efd7d34d732014f4f0d7f65d359f064220e32965849ac2ab35eb734897"}
Sep 17 22:00:02.653229 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.653103    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zzlwg" event={"ID":"6f2123ef-58e4-400e-8708-622b61aef80b","Type":"ContainerStarted","Data":"428e3c020e43e181d008ed4134cf6eb82114463413f46945b4b734fc1636a746"}
Sep 17 22:00:02.654330 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.654304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9","Type":"ContainerStarted","Data":"b139ec511374d49b9b7e33cd4cbd8a4f3c98ac75a1eac0ad476175de96ba86db"}
Sep 17 22:00:02.670555 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.670510    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-j8h4q" podStartSLOduration=2.060124256 podStartE2EDuration="3.670493543s" totalImagesPullingTime="1.610369287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:02.668399577 +0000 UTC m=+101.751987549" watchObservedRunningTime="2026-09-17 22:00:02.670493543 +0000 UTC m=+101.754081493"
Sep 17 22:00:02.686760 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.686716    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zzlwg" podStartSLOduration=2.284987644 podStartE2EDuration="3.686703726s" totalImagesPullingTime="1.329162775s" totalInitContainerRuntime="72.553307ms" isStatefulPod=true podCreationTimestamp="2026-09-17 21:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:02.685622543 +0000 UTC m=+101.769210492" watchObservedRunningTime="2026-09-17 22:00:02.686703726 +0000 UTC m=+101.770291675"
Sep 17 22:00:02.796374 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.796334    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:02.952070 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:02.951995    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:03.815697 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.815660    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"]
Sep 17 22:00:03.819065 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.819042    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.821763 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.821742    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 22:00:03.822050 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.822030    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 22:00:03.822894 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.822877    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 22:00:03.823317 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.823298    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5h7kfqpr098d3\""
Sep 17 22:00:03.823403 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.823300    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 22:00:03.823403 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.823303    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-l47j6\""
Sep 17 22:00:03.836713 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.836685    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"]
Sep 17 22:00:03.861013 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.860987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.861112 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.861038    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-client-ca-bundle\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.861112 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.861076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-secret-metrics-server-tls\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.861112 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.861107    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-audit-log\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.861437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.861129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75b2\" (UniqueName: \"kubernetes.io/projected/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-kube-api-access-g75b2\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.861437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.861245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-secret-metrics-server-client-certs\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.861437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.861370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-metrics-server-audit-profiles\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.962206 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.962059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-secret-metrics-server-client-certs\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.962206 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.962119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-metrics-server-audit-profiles\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.962376 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.962213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.962376 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.962252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-client-ca-bundle\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.962376 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.962290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-secret-metrics-server-tls\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.962376 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.962331    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-audit-log\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.962801 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.962731    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-audit-log\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.963067 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.963043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.963192 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.963104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75b2\" (UniqueName: \"kubernetes.io/projected/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-kube-api-access-g75b2\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.964153 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.963651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-metrics-server-audit-profiles\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.965702 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.965681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-secret-metrics-server-client-certs\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.965780 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.965699    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-client-ca-bundle\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.966617 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.966593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-secret-metrics-server-tls\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:03.971452 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:03.971429    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75b2\" (UniqueName: \"kubernetes.io/projected/e0252fd0-162f-4fc4-a6b2-df0acb473f5f-kube-api-access-g75b2\") pod \"metrics-server-6f5fb6c5b4-9bsjc\" (UID: \"e0252fd0-162f-4fc4-a6b2-df0acb473f5f\") " pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:04.128882 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.128847    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:04.189219 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.189079    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:04.200765 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.200685    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"]
Sep 17 22:00:04.205683 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.205660    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"
Sep 17 22:00:04.210182 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.208258    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m6clm\""
Sep 17 22:00:04.210182 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.208826    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 22:00:04.213759 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.213712    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"]
Sep 17 22:00:04.247627 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.247601    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:04.266210 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.266184    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzkj\" (UniqueName: \"kubernetes.io/projected/d1844c7f-fe30-43a7-96d0-1e724a84c003-kube-api-access-rqzkj\") pod \"monitoring-plugin-5984459ff8-vc9mj\" (UID: \"d1844c7f-fe30-43a7-96d0-1e724a84c003\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"
Sep 17 22:00:04.266335 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.266311    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1844c7f-fe30-43a7-96d0-1e724a84c003-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vc9mj\" (UID: \"d1844c7f-fe30-43a7-96d0-1e724a84c003\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"
Sep 17 22:00:04.367684 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.367643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzkj\" (UniqueName: \"kubernetes.io/projected/d1844c7f-fe30-43a7-96d0-1e724a84c003-kube-api-access-rqzkj\") pod \"monitoring-plugin-5984459ff8-vc9mj\" (UID: \"d1844c7f-fe30-43a7-96d0-1e724a84c003\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"
Sep 17 22:00:04.367851 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.367734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1844c7f-fe30-43a7-96d0-1e724a84c003-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vc9mj\" (UID: \"d1844c7f-fe30-43a7-96d0-1e724a84c003\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"
Sep 17 22:00:04.367851 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:00:04.367825    2583 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 22:00:04.367941 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:00:04.367879    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1844c7f-fe30-43a7-96d0-1e724a84c003-monitoring-plugin-cert podName:d1844c7f-fe30-43a7-96d0-1e724a84c003 nodeName:}" failed. No retries permitted until 2026-09-17 22:00:04.867865334 +0000 UTC m=+103.951453261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d1844c7f-fe30-43a7-96d0-1e724a84c003-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-vc9mj" (UID: "d1844c7f-fe30-43a7-96d0-1e724a84c003") : secret "monitoring-plugin-cert" not found
Sep 17 22:00:04.376595 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.376572    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzkj\" (UniqueName: \"kubernetes.io/projected/d1844c7f-fe30-43a7-96d0-1e724a84c003-kube-api-access-rqzkj\") pod \"monitoring-plugin-5984459ff8-vc9mj\" (UID: \"d1844c7f-fe30-43a7-96d0-1e724a84c003\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"
Sep 17 22:00:04.416697 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.416675    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"]
Sep 17 22:00:04.419230 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:00:04.419208    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0252fd0_162f_4fc4_a6b2_df0acb473f5f.slice/crio-954126a29dd03fd942ef797e2d6afed907286beb2055d563ade5a75086bb7529 WatchSource:0}: Error finding container 954126a29dd03fd942ef797e2d6afed907286beb2055d563ade5a75086bb7529: Status 404 returned error can't find the container with id 954126a29dd03fd942ef797e2d6afed907286beb2055d563ade5a75086bb7529
Sep 17 22:00:04.454739 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.454717    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:04.665441 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.665406    2583 generic.go:356] "Generic (PLEG): container finished" podID="390aa30d-b0eb-4036-9c3b-1f8520b3e7a9" containerID="ff9781ab339a9c610bd04a24e46a93a96bd7b2bdc41f47922c9d83bfc5995ec6" exitCode=0
Sep 17 22:00:04.665628 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.665490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9","Type":"ContainerDied","Data":"ff9781ab339a9c610bd04a24e46a93a96bd7b2bdc41f47922c9d83bfc5995ec6"}
Sep 17 22:00:04.666849 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.666791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc" event={"ID":"e0252fd0-162f-4fc4-a6b2-df0acb473f5f","Type":"ContainerStarted","Data":"954126a29dd03fd942ef797e2d6afed907286beb2055d563ade5a75086bb7529"}
Sep 17 22:00:04.668767 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.668711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2x2z9" event={"ID":"370fad22-3b0a-489b-a3f9-f91ff54de074","Type":"ContainerStarted","Data":"872eed7686938f71a96b424783755ad1fe3f4200a5e13133673354e90595ef2d"}
Sep 17 22:00:04.668767 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.668751    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2x2z9" event={"ID":"370fad22-3b0a-489b-a3f9-f91ff54de074","Type":"ContainerStarted","Data":"97479935f8b2911b29a2cb6f7c40a7193366aee416c94aeaef9c076b34c92936"}
Sep 17 22:00:04.668889 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.668840    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2x2z9"
Sep 17 22:00:04.748001 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.747946    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2x2z9" podStartSLOduration=65.726029477 podStartE2EDuration="1m8.747929739s" totalImagesPullingTime="3.021900262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:04.747218192 +0000 UTC m=+103.830806161" watchObservedRunningTime="2026-09-17 22:00:04.747929739 +0000 UTC m=+103.831517688"
Sep 17 22:00:04.871174 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.871125    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1844c7f-fe30-43a7-96d0-1e724a84c003-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vc9mj\" (UID: \"d1844c7f-fe30-43a7-96d0-1e724a84c003\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"
Sep 17 22:00:04.873762 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:04.873733    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1844c7f-fe30-43a7-96d0-1e724a84c003-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-vc9mj\" (UID: \"d1844c7f-fe30-43a7-96d0-1e724a84c003\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"
Sep 17 22:00:05.122112 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:05.122035    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"
Sep 17 22:00:05.411214 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:05.411111    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"]
Sep 17 22:00:05.418866 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:00:05.418819    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1844c7f_fe30_43a7_96d0_1e724a84c003.slice/crio-64aabe99f0b106ea76f2ed7e0529eb0a365792e38bcabf71335184fdc0d3338e WatchSource:0}: Error finding container 64aabe99f0b106ea76f2ed7e0529eb0a365792e38bcabf71335184fdc0d3338e: Status 404 returned error can't find the container with id 64aabe99f0b106ea76f2ed7e0529eb0a365792e38bcabf71335184fdc0d3338e
Sep 17 22:00:05.527266 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:05.527237    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q8xvj"
Sep 17 22:00:05.532951 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:05.532907    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:05.634584 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:05.634551    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:05.676007 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:05.675924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj" event={"ID":"d1844c7f-fe30-43a7-96d0-1e724a84c003","Type":"ContainerStarted","Data":"64aabe99f0b106ea76f2ed7e0529eb0a365792e38bcabf71335184fdc0d3338e"}
Sep 17 22:00:05.720072 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:05.719725    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:06.475368 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.475333    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69d59dcf6-7zmc4"]
Sep 17 22:00:06.490377 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.490351    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d59dcf6-7zmc4"]
Sep 17 22:00:06.490540 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.490456    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.493513 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.493489    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 22:00:06.493644 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.493537    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 22:00:06.493644 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.493622    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 22:00:06.494435 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.494347    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 22:00:06.494800 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.494732    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 22:00:06.494909 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.494861    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 22:00:06.494976 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.494870    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5m25g\""
Sep 17 22:00:06.494976 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.494938    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 22:00:06.502284 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.501310    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 22:00:06.586812 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.586770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-trusted-ca-bundle\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.586984 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.586820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmsmk\" (UniqueName: \"kubernetes.io/projected/837883fb-9efc-4cdd-963d-cac7345eed9b-kube-api-access-mmsmk\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.586984 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.586918    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-serving-cert\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.586984 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.586958    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-console-config\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.587125 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.587009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-oauth-serving-cert\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.587125 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.587062    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-service-ca\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.587125 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.587099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-oauth-config\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.688428 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.688393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-service-ca\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.688615 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.688434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-oauth-config\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.688615 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.688506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-trusted-ca-bundle\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.688615 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.688526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmsmk\" (UniqueName: \"kubernetes.io/projected/837883fb-9efc-4cdd-963d-cac7345eed9b-kube-api-access-mmsmk\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.688615 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.688564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-serving-cert\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.688615 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.688587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-console-config\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.688876 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.688660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-oauth-serving-cert\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.689335 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.689303    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-service-ca\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.689454 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.689427    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-oauth-serving-cert\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.689454 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.689446    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-console-config\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.689573 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.689544    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-trusted-ca-bundle\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.691382 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.691360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-oauth-config\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.691488 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.691360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-serving-cert\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.698537 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.698492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmsmk\" (UniqueName: \"kubernetes.io/projected/837883fb-9efc-4cdd-963d-cac7345eed9b-kube-api-access-mmsmk\") pod \"console-69d59dcf6-7zmc4\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") " pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:06.807622 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:06.807544    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:07.499699 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:07.499594    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d59dcf6-7zmc4"]
Sep 17 22:00:07.505555 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:00:07.505530    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837883fb_9efc_4cdd_963d_cac7345eed9b.slice/crio-c9c3c0c2579a1e2ca204e218fec60c8c52592c4e62bd83ad324bcf664e3d681a WatchSource:0}: Error finding container c9c3c0c2579a1e2ca204e218fec60c8c52592c4e62bd83ad324bcf664e3d681a: Status 404 returned error can't find the container with id c9c3c0c2579a1e2ca204e218fec60c8c52592c4e62bd83ad324bcf664e3d681a
Sep 17 22:00:07.689721 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:07.689687    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d59dcf6-7zmc4" event={"ID":"837883fb-9efc-4cdd-963d-cac7345eed9b","Type":"ContainerStarted","Data":"c9c3c0c2579a1e2ca204e218fec60c8c52592c4e62bd83ad324bcf664e3d681a"}
Sep 17 22:00:07.693238 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:07.693172    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9","Type":"ContainerStarted","Data":"bbdd0619dd3c15e89e0dee1eaf6f0d6c9d830390c8ed0f13a5e281f5abbe13ea"}
Sep 17 22:00:07.693370 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:07.693257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9","Type":"ContainerStarted","Data":"d4944ec3cb5721958b3736b34e38c1b177479da34637bdfb72e1cc53b3cd0e90"}
Sep 17 22:00:07.693370 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:07.693301    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9","Type":"ContainerStarted","Data":"3ba99342dae3b2a3f44d92a52e3731a46cba1624e56c81ac87d352c814ce9dc1"}
Sep 17 22:00:07.694858 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:07.694834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc" event={"ID":"e0252fd0-162f-4fc4-a6b2-df0acb473f5f","Type":"ContainerStarted","Data":"f9c7cf2f9da3ca3dce608257b4d68967ee2b219ef4f589d24a42ac3eb3aa74b2"}
Sep 17 22:00:07.715190 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:07.715111    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc" podStartSLOduration=1.915042169 podStartE2EDuration="4.715093408s" totalImagesPullingTime="2.800051239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:07.713556595 +0000 UTC m=+106.797144568" watchObservedRunningTime="2026-09-17 22:00:07.715093408 +0000 UTC m=+106.798681358"
Sep 17 22:00:07.776435 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:07.776351    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:07.833953 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:07.833919    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:07.981466 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:07.980110    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:08.044946 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:08.044870    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:08.107735 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:08.107708    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:08.704017 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:08.703921    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj" event={"ID":"d1844c7f-fe30-43a7-96d0-1e724a84c003","Type":"ContainerStarted","Data":"6e4d0df5453b67d466e188afa1f3b99cd84ff6254928a86bb64d77bace119741"}
Sep 17 22:00:08.704691 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:08.704670    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"
Sep 17 22:00:08.711325 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:08.711105    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj"
Sep 17 22:00:08.713311 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:08.713155    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9","Type":"ContainerStarted","Data":"14341e3140a949c235052305b2be44c0b278b25b0a398a56b6cb56b68af5f5e9"}
Sep 17 22:00:08.713311 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:08.713235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9","Type":"ContainerStarted","Data":"90d8a5a5c0a5bffabbc37416dc64020148bba8a93783a06ada9981da672d83b9"}
Sep 17 22:00:08.737179 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:08.736087    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-vc9mj" podStartSLOduration=1.8014997400000001 podStartE2EDuration="4.736060217s" totalImagesPullingTime="2.934560477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:08.720822803 +0000 UTC m=+107.804410777" watchObservedRunningTime="2026-09-17 22:00:08.736060217 +0000 UTC m=+107.819648167"
Sep 17 22:00:09.324204 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:09.324169    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:09.390168 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:09.390095    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:12.734399 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:12.734361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"390aa30d-b0eb-4036-9c3b-1f8520b3e7a9","Type":"ContainerStarted","Data":"e4ede48b6a5bd630e9f882a1d7f6bb7ee7af016107fed13491654f1db41197a5"}
Sep 17 22:00:12.735835 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:12.735811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d59dcf6-7zmc4" event={"ID":"837883fb-9efc-4cdd-963d-cac7345eed9b","Type":"ContainerStarted","Data":"24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f"}
Sep 17 22:00:12.764504 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:12.764453    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.270836338 podStartE2EDuration="12.764436496s" totalImagesPullingTime="8.476979501s" totalInitContainerRuntime="16.620657ms" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:12.763484644 +0000 UTC m=+111.847072613" watchObservedRunningTime="2026-09-17 22:00:12.764436496 +0000 UTC m=+111.848024495"
Sep 17 22:00:12.786082 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:12.786023    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69d59dcf6-7zmc4" podStartSLOduration=2.196979188 podStartE2EDuration="6.786006374s" totalImagesPullingTime="4.589027186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:12.785630348 +0000 UTC m=+111.869218299" watchObservedRunningTime="2026-09-17 22:00:12.786006374 +0000 UTC m=+111.869594322"
Sep 17 22:00:14.678518 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:14.678486    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2x2z9"
Sep 17 22:00:16.243270 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:16.243235    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:16.344586 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:16.344549    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:16.808118 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:16.808078    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:16.808307 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:16.808186    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:16.812700 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:16.812678    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:17.515782 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:17.515750    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:17.616704 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:17.616675    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:17.759709 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:17.759686    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:17.895641 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:17.895599    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:17.996458 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:17.996413    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:19.977444 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:19.977416    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:20.037531 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:20.037504    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:22.031741 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:22.031707    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d59dcf6-7zmc4"]
Sep 17 22:00:22.148930 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:22.148899    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:22.266474 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:22.266434    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:22.350399 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:22.350308    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:22.810762 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:22.810732    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:22.915253 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:22.915217    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:24.129441 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:24.129405    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:24.129886 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:24.129511    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:25.267757 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:25.267711    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:25.364248 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:25.364208    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:26.606631 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:26.606579    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:26.707737 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:26.707712    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:28.423466 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:28.423395    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:28.493466 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:28.493438    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:29.613593 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:29.613556    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:29.670624 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:29.670596    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:30.807051 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:30.807019    2583 generic.go:356] "Generic (PLEG): container finished" podID="0a9a9e08-09a2-4062-9b80-d293592d8df2" containerID="1fd235404a8136fb233a8f95244208719c34509301b14d7bcf45491f723ff255" exitCode=0
Sep 17 22:00:30.807438 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:30.807087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx" event={"ID":"0a9a9e08-09a2-4062-9b80-d293592d8df2","Type":"ContainerDied","Data":"1fd235404a8136fb233a8f95244208719c34509301b14d7bcf45491f723ff255"}
Sep 17 22:00:30.807477 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:30.807460    2583 scope.go:117] "RemoveContainer" containerID="1fd235404a8136fb233a8f95244208719c34509301b14d7bcf45491f723ff255"
Sep 17 22:00:31.812583 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:31.812554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-mhdkx" event={"ID":"0a9a9e08-09a2-4062-9b80-d293592d8df2","Type":"ContainerStarted","Data":"4073652fe6e9a2db578bf8523f91916d5a9b7e247d1520894c4d9cc8f61be50b"}
Sep 17 22:00:32.140518 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:32.140484    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:32.206528 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:32.206500    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:32.919027 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:32.918985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 22:00:32.921409 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:32.921385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cc2ec45-0451-46da-aa90-214fd10d39bd-metrics-certs\") pod \"network-metrics-daemon-n64gh\" (UID: \"6cc2ec45-0451-46da-aa90-214fd10d39bd\") " pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 22:00:33.004313 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:33.004283    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pzq2x\""
Sep 17 22:00:33.012252 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:33.012234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n64gh"
Sep 17 22:00:33.277155 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:33.277080    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n64gh"]
Sep 17 22:00:33.279414 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:00:33.279378    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc2ec45_0451_46da_aa90_214fd10d39bd.slice/crio-fa502385409095ff84122eec6afe36d1f1c3802fcf4518c180d94dd77c6a883b WatchSource:0}: Error finding container fa502385409095ff84122eec6afe36d1f1c3802fcf4518c180d94dd77c6a883b: Status 404 returned error can't find the container with id fa502385409095ff84122eec6afe36d1f1c3802fcf4518c180d94dd77c6a883b
Sep 17 22:00:33.821871 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:33.821834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n64gh" event={"ID":"6cc2ec45-0451-46da-aa90-214fd10d39bd","Type":"ContainerStarted","Data":"fa502385409095ff84122eec6afe36d1f1c3802fcf4518c180d94dd77c6a883b"}
Sep 17 22:00:34.827914 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:34.827881    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n64gh" event={"ID":"6cc2ec45-0451-46da-aa90-214fd10d39bd","Type":"ContainerStarted","Data":"74da5cdbcba5815577e3466432587a459ccf9b5e2d0c18065f2e0301a4c851c4"}
Sep 17 22:00:34.828330 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:34.827934    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n64gh" event={"ID":"6cc2ec45-0451-46da-aa90-214fd10d39bd","Type":"ContainerStarted","Data":"1e4f7a5957dc46d45da7804d25400742c46d93b3ff33dda975c7b3ff1cf0862f"}
Sep 17 22:00:34.850422 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:34.850370    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n64gh" podStartSLOduration=130.969664479 podStartE2EDuration="2m11.85035218s" totalImagesPullingTime="880.687701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:00:34.849420069 +0000 UTC m=+133.933008018" watchObservedRunningTime="2026-09-17 22:00:34.85035218 +0000 UTC m=+133.933940128"
Sep 17 22:00:34.981924 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:34.981889    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:35.054688 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:35.054652    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:44.135423 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:44.135394    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:44.139921 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:44.139897    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6f5fb6c5b4-9bsjc"
Sep 17 22:00:47.053261 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.053195    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69d59dcf6-7zmc4" podUID="837883fb-9efc-4cdd-963d-cac7345eed9b" containerName="console" containerID="cri-o://24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f" gracePeriod=15
Sep 17 22:00:47.303439 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.303388    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:47.447008 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.446964    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-serving-cert\") pod \"837883fb-9efc-4cdd-963d-cac7345eed9b\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") "
Sep 17 22:00:47.447008 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447007    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-console-config\") pod \"837883fb-9efc-4cdd-963d-cac7345eed9b\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") "
Sep 17 22:00:47.447294 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447037    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmsmk\" (UniqueName: \"kubernetes.io/projected/837883fb-9efc-4cdd-963d-cac7345eed9b-kube-api-access-mmsmk\") pod \"837883fb-9efc-4cdd-963d-cac7345eed9b\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") "
Sep 17 22:00:47.447294 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447060    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-oauth-serving-cert\") pod \"837883fb-9efc-4cdd-963d-cac7345eed9b\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") "
Sep 17 22:00:47.447294 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447201    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-trusted-ca-bundle\") pod \"837883fb-9efc-4cdd-963d-cac7345eed9b\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") "
Sep 17 22:00:47.447447 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447297    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-service-ca\") pod \"837883fb-9efc-4cdd-963d-cac7345eed9b\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") "
Sep 17 22:00:47.447447 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447351    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-oauth-config\") pod \"837883fb-9efc-4cdd-963d-cac7345eed9b\" (UID: \"837883fb-9efc-4cdd-963d-cac7345eed9b\") "
Sep 17 22:00:47.447558 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447488    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-console-config" (OuterVolumeSpecName: "console-config") pod "837883fb-9efc-4cdd-963d-cac7345eed9b" (UID: "837883fb-9efc-4cdd-963d-cac7345eed9b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:47.447558 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447520    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "837883fb-9efc-4cdd-963d-cac7345eed9b" (UID: "837883fb-9efc-4cdd-963d-cac7345eed9b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:47.447661 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447638    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "837883fb-9efc-4cdd-963d-cac7345eed9b" (UID: "837883fb-9efc-4cdd-963d-cac7345eed9b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:47.447721 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447696    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-console-config\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:00:47.447721 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447699    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-service-ca" (OuterVolumeSpecName: "service-ca") pod "837883fb-9efc-4cdd-963d-cac7345eed9b" (UID: "837883fb-9efc-4cdd-963d-cac7345eed9b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:00:47.447721 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.447714    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-oauth-serving-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:00:47.449506 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.449481    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837883fb-9efc-4cdd-963d-cac7345eed9b-kube-api-access-mmsmk" (OuterVolumeSpecName: "kube-api-access-mmsmk") pod "837883fb-9efc-4cdd-963d-cac7345eed9b" (UID: "837883fb-9efc-4cdd-963d-cac7345eed9b"). InnerVolumeSpecName "kube-api-access-mmsmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:00:47.449596 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.449506    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "837883fb-9efc-4cdd-963d-cac7345eed9b" (UID: "837883fb-9efc-4cdd-963d-cac7345eed9b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:47.449596 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.449544    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "837883fb-9efc-4cdd-963d-cac7345eed9b" (UID: "837883fb-9efc-4cdd-963d-cac7345eed9b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:00:47.548527 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.548493    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-oauth-config\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:00:47.548527 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.548520    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/837883fb-9efc-4cdd-963d-cac7345eed9b-console-serving-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:00:47.548527 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.548533    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmsmk\" (UniqueName: \"kubernetes.io/projected/837883fb-9efc-4cdd-963d-cac7345eed9b-kube-api-access-mmsmk\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:00:47.548729 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.548546    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-trusted-ca-bundle\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:00:47.548729 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.548559    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/837883fb-9efc-4cdd-963d-cac7345eed9b-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:00:47.888565 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.888525    2583 generic.go:356] "Generic (PLEG): container finished" podID="837883fb-9efc-4cdd-963d-cac7345eed9b" containerID="24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f" exitCode=0
Sep 17 22:00:47.888763 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.888572    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d59dcf6-7zmc4"
Sep 17 22:00:47.888763 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.888615    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d59dcf6-7zmc4" event={"ID":"837883fb-9efc-4cdd-963d-cac7345eed9b","Type":"ContainerDied","Data":"24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f"}
Sep 17 22:00:47.888763 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.888669    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d59dcf6-7zmc4" event={"ID":"837883fb-9efc-4cdd-963d-cac7345eed9b","Type":"ContainerDied","Data":"c9c3c0c2579a1e2ca204e218fec60c8c52592c4e62bd83ad324bcf664e3d681a"}
Sep 17 22:00:47.888763 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.888695    2583 scope.go:117] "RemoveContainer" containerID="24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f"
Sep 17 22:00:47.897405 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.897382    2583 scope.go:117] "RemoveContainer" containerID="24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f"
Sep 17 22:00:47.897679 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:00:47.897658    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f\": container with ID starting with 24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f not found: ID does not exist" containerID="24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f"
Sep 17 22:00:47.897724 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.897687    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f"} err="failed to get container status \"24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f\": rpc error: code = NotFound desc = could not find container \"24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f\": container with ID starting with 24d24ca3370b67528e1b5ef9659862bad32b833c88f8036ec7a79c1702e13b6f not found: ID does not exist"
Sep 17 22:00:47.908777 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.908752    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d59dcf6-7zmc4"]
Sep 17 22:00:47.912444 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:47.912424    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69d59dcf6-7zmc4"]
Sep 17 22:00:49.186355 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:49.186328    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837883fb-9efc-4cdd-963d-cac7345eed9b" path="/var/lib/kubelet/pods/837883fb-9efc-4cdd-963d-cac7345eed9b/volumes"
Sep 17 22:00:52.780151 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:52.780098    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:00:52.837704 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:00:52.837667    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:15.540514 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:15.540432    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:15.611895 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:15.611862    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:26.841728 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:26.841688    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:26.903506 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:26.903473    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:27.534573 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:27.534535    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:27.584190 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:27.584158    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:37.405290 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:37.405254    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:37.457282 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:37.457249    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:37.512661 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:37.512629    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:37.784749 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:37.784654    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:37.884580 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:37.884547    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:54.484450 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:54.484413    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:01:54.543565 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:01:54.543518    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:02:02.118639 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:02:02.118591    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:02:02.176034 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:02:02.176004    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:02:05.176322 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:02:05.176286    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:02:37.720061 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:02:37.719978    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:02:37.820705 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:02:37.820669    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:02:37.925111 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:02:37.925078    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:02:59.729469 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:02:59.729435    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:02:59.828458 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:02:59.828415    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:01.390421 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.390387    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"]
Sep 17 22:03:01.390796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.390722    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837883fb-9efc-4cdd-963d-cac7345eed9b" containerName="console"
Sep 17 22:03:01.390796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.390734    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="837883fb-9efc-4cdd-963d-cac7345eed9b" containerName="console"
Sep 17 22:03:01.390796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.390783    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="837883fb-9efc-4cdd-963d-cac7345eed9b" containerName="console"
Sep 17 22:03:01.392783 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.392767    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"
Sep 17 22:03:01.396400 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.396378    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 22:03:01.396901 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.396888    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-7m9k2\""
Sep 17 22:03:01.396972 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.396937    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 22:03:01.397531 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.397516    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 22:03:01.412799 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.412772    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"]
Sep 17 22:03:01.441015 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.440982    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/981bcf6e-ceec-4652-afe1-400d5d2584c3-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djjcn\" (UID: \"981bcf6e-ceec-4652-afe1-400d5d2584c3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"
Sep 17 22:03:01.441217 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.441078    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l26zn\" (UniqueName: \"kubernetes.io/projected/981bcf6e-ceec-4652-afe1-400d5d2584c3-kube-api-access-l26zn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djjcn\" (UID: \"981bcf6e-ceec-4652-afe1-400d5d2584c3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"
Sep 17 22:03:01.541669 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.541627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l26zn\" (UniqueName: \"kubernetes.io/projected/981bcf6e-ceec-4652-afe1-400d5d2584c3-kube-api-access-l26zn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djjcn\" (UID: \"981bcf6e-ceec-4652-afe1-400d5d2584c3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"
Sep 17 22:03:01.541669 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.541675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/981bcf6e-ceec-4652-afe1-400d5d2584c3-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djjcn\" (UID: \"981bcf6e-ceec-4652-afe1-400d5d2584c3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"
Sep 17 22:03:01.544014 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.543980    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/981bcf6e-ceec-4652-afe1-400d5d2584c3-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djjcn\" (UID: \"981bcf6e-ceec-4652-afe1-400d5d2584c3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"
Sep 17 22:03:01.552690 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.552664    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l26zn\" (UniqueName: \"kubernetes.io/projected/981bcf6e-ceec-4652-afe1-400d5d2584c3-kube-api-access-l26zn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-djjcn\" (UID: \"981bcf6e-ceec-4652-afe1-400d5d2584c3\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"
Sep 17 22:03:01.702938 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.702839    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"
Sep 17 22:03:01.975363 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:01.975339    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"]
Sep 17 22:03:01.978159 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:03:01.978101    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981bcf6e_ceec_4652_afe1_400d5d2584c3.slice/crio-4ceda6eb8d2f2649d1c27f38af1277cdebac584a1546649082447b8bcddac546 WatchSource:0}: Error finding container 4ceda6eb8d2f2649d1c27f38af1277cdebac584a1546649082447b8bcddac546: Status 404 returned error can't find the container with id 4ceda6eb8d2f2649d1c27f38af1277cdebac584a1546649082447b8bcddac546
Sep 17 22:03:02.420568 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:02.420528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn" event={"ID":"981bcf6e-ceec-4652-afe1-400d5d2584c3","Type":"ContainerStarted","Data":"4ceda6eb8d2f2649d1c27f38af1277cdebac584a1546649082447b8bcddac546"}
Sep 17 22:03:02.545263 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:02.545232    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:05.436970 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:05.436864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn" event={"ID":"981bcf6e-ceec-4652-afe1-400d5d2584c3","Type":"ContainerStarted","Data":"3e3c9104e5fa3df94106a4d25182d791282d8752446ab645640ddfa0ec543b3d"}
Sep 17 22:03:05.436970 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:05.436942    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"
Sep 17 22:03:05.457775 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:05.457719    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn" podStartSLOduration=1.25167238 podStartE2EDuration="4.457701156s" totalImagesPullingTime="3.206028776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:05.45578903 +0000 UTC m=+284.539377027" watchObservedRunningTime="2026-09-17 22:03:05.457701156 +0000 UTC m=+284.541289106"
Sep 17 22:03:05.721742 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:05.721654    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:06.000360 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.000266    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:06.180619 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.180573    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:06.277857 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.277770    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:06.379581 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.379545    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"]
Sep 17 22:03:06.382020 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.382001    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:06.386256 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.386233    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-64shg\""
Sep 17 22:03:06.386756 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.386732    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 22:03:06.387270 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.387253    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 22:03:06.396433 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.396408    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"]
Sep 17 22:03:06.487360 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.487326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gtgkw\" (UID: \"3dc861b2-f144-410f-a853-3dc1a277f4b5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:06.487360 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.487363    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ttf\" (UniqueName: \"kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-kube-api-access-p2ttf\") pod \"keda-metrics-apiserver-77857b7b98-gtgkw\" (UID: \"3dc861b2-f144-410f-a853-3dc1a277f4b5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:06.487791 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.487390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3dc861b2-f144-410f-a853-3dc1a277f4b5-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gtgkw\" (UID: \"3dc861b2-f144-410f-a853-3dc1a277f4b5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:06.588812 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.588726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gtgkw\" (UID: \"3dc861b2-f144-410f-a853-3dc1a277f4b5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:06.588812 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.588763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ttf\" (UniqueName: \"kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-kube-api-access-p2ttf\") pod \"keda-metrics-apiserver-77857b7b98-gtgkw\" (UID: \"3dc861b2-f144-410f-a853-3dc1a277f4b5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:06.588812 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.588792    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3dc861b2-f144-410f-a853-3dc1a277f4b5-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gtgkw\" (UID: \"3dc861b2-f144-410f-a853-3dc1a277f4b5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:06.589031 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:03:06.588890    2583 secret.go:281] references non-existent secret key: tls.crt
Sep 17 22:03:06.589031 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:03:06.588918    2583 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 17 22:03:06.589031 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:03:06.588938    2583 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 17 22:03:06.589031 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:03:06.588958    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 17 22:03:06.589031 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:03:06.589014    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-certificates podName:3dc861b2-f144-410f-a853-3dc1a277f4b5 nodeName:}" failed. No retries permitted until 2026-09-17 22:03:07.088994179 +0000 UTC m=+286.172582114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-certificates") pod "keda-metrics-apiserver-77857b7b98-gtgkw" (UID: "3dc861b2-f144-410f-a853-3dc1a277f4b5") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 17 22:03:06.589236 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.589127    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3dc861b2-f144-410f-a853-3dc1a277f4b5-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-gtgkw\" (UID: \"3dc861b2-f144-410f-a853-3dc1a277f4b5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:06.600518 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:06.600491    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ttf\" (UniqueName: \"kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-kube-api-access-p2ttf\") pod \"keda-metrics-apiserver-77857b7b98-gtgkw\" (UID: \"3dc861b2-f144-410f-a853-3dc1a277f4b5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:07.093800 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.093761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gtgkw\" (UID: \"3dc861b2-f144-410f-a853-3dc1a277f4b5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:07.093991 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:03:07.093929    2583 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 17 22:03:07.093991 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:03:07.093959    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw: secret "keda-metrics-apiserver-certs" not found
Sep 17 22:03:07.094117 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:03:07.094027    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-certificates podName:3dc861b2-f144-410f-a853-3dc1a277f4b5 nodeName:}" failed. No retries permitted until 2026-09-17 22:03:08.094006212 +0000 UTC m=+287.177594154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-certificates") pod "keda-metrics-apiserver-77857b7b98-gtgkw" (UID: "3dc861b2-f144-410f-a853-3dc1a277f4b5") : secret "keda-metrics-apiserver-certs" not found
Sep 17 22:03:07.116856 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.116819    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:07.131318 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.131275    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-qttx6"]
Sep 17 22:03:07.134360 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.134342    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-qttx6"
Sep 17 22:03:07.137749 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.137723    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 22:03:07.148328 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.148300    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-qttx6"]
Sep 17 22:03:07.194865 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.194820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e-certificates\") pod \"keda-admission-6c4d864b86-qttx6\" (UID: \"c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e\") " pod="openshift-keda/keda-admission-6c4d864b86-qttx6"
Sep 17 22:03:07.195057 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.194955    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6wx7\" (UniqueName: \"kubernetes.io/projected/c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e-kube-api-access-s6wx7\") pod \"keda-admission-6c4d864b86-qttx6\" (UID: \"c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e\") " pod="openshift-keda/keda-admission-6c4d864b86-qttx6"
Sep 17 22:03:07.208380 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.208333    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:07.295865 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.295827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e-certificates\") pod \"keda-admission-6c4d864b86-qttx6\" (UID: \"c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e\") " pod="openshift-keda/keda-admission-6c4d864b86-qttx6"
Sep 17 22:03:07.296036 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.295913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6wx7\" (UniqueName: \"kubernetes.io/projected/c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e-kube-api-access-s6wx7\") pod \"keda-admission-6c4d864b86-qttx6\" (UID: \"c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e\") " pod="openshift-keda/keda-admission-6c4d864b86-qttx6"
Sep 17 22:03:07.296036 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:03:07.295997    2583 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 22:03:07.296036 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:03:07.296023    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-qttx6: secret "keda-admission-webhooks-certs" not found
Sep 17 22:03:07.296153 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:03:07.296085    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e-certificates podName:c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e nodeName:}" failed. No retries permitted until 2026-09-17 22:03:07.79606519 +0000 UTC m=+286.879653131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e-certificates") pod "keda-admission-6c4d864b86-qttx6" (UID: "c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e") : secret "keda-admission-webhooks-certs" not found
Sep 17 22:03:07.308407 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.308379    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6wx7\" (UniqueName: \"kubernetes.io/projected/c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e-kube-api-access-s6wx7\") pod \"keda-admission-6c4d864b86-qttx6\" (UID: \"c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e\") " pod="openshift-keda/keda-admission-6c4d864b86-qttx6"
Sep 17 22:03:07.352795 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.352695    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:07.471920 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.471882    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:07.559468 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.559429    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:07.658797 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.658744    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:07.801124 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.801086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e-certificates\") pod \"keda-admission-6c4d864b86-qttx6\" (UID: \"c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e\") " pod="openshift-keda/keda-admission-6c4d864b86-qttx6"
Sep 17 22:03:07.803729 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.803700    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:07.803835 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.803709    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e-certificates\") pod \"keda-admission-6c4d864b86-qttx6\" (UID: \"c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e\") " pod="openshift-keda/keda-admission-6c4d864b86-qttx6"
Sep 17 22:03:07.906089 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:07.906054    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:08.045182 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:08.045063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-qttx6"
Sep 17 22:03:08.104824 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:08.104789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gtgkw\" (UID: \"3dc861b2-f144-410f-a853-3dc1a277f4b5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:08.108968 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:08.108910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3dc861b2-f144-410f-a853-3dc1a277f4b5-certificates\") pod \"keda-metrics-apiserver-77857b7b98-gtgkw\" (UID: \"3dc861b2-f144-410f-a853-3dc1a277f4b5\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:08.192575 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:08.192541    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:08.318653 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:08.318557    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-qttx6"]
Sep 17 22:03:08.321273 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:03:08.321244    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b6ff81_a886_4fe4_a1a1_4c4d43f6068e.slice/crio-ae6fea4a08efa5d3d0ab9d7f97116a5a0bfe13ab6701827790f8720a64f8bd4f WatchSource:0}: Error finding container ae6fea4a08efa5d3d0ab9d7f97116a5a0bfe13ab6701827790f8720a64f8bd4f: Status 404 returned error can't find the container with id ae6fea4a08efa5d3d0ab9d7f97116a5a0bfe13ab6701827790f8720a64f8bd4f
Sep 17 22:03:08.451844 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:08.451810    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-qttx6" event={"ID":"c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e","Type":"ContainerStarted","Data":"ae6fea4a08efa5d3d0ab9d7f97116a5a0bfe13ab6701827790f8720a64f8bd4f"}
Sep 17 22:03:08.462209 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:08.461311    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"]
Sep 17 22:03:08.468131 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:03:08.468102    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc861b2_f144_410f_a853_3dc1a277f4b5.slice/crio-472e68d6ca552573c9a37efd3b2fc2d01e6e3a279377548328c03c65852302e6 WatchSource:0}: Error finding container 472e68d6ca552573c9a37efd3b2fc2d01e6e3a279377548328c03c65852302e6: Status 404 returned error can't find the container with id 472e68d6ca552573c9a37efd3b2fc2d01e6e3a279377548328c03c65852302e6
Sep 17 22:03:09.458659 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:09.458609    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw" event={"ID":"3dc861b2-f144-410f-a853-3dc1a277f4b5","Type":"ContainerStarted","Data":"472e68d6ca552573c9a37efd3b2fc2d01e6e3a279377548328c03c65852302e6"}
Sep 17 22:03:10.465677 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:10.465638    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-qttx6" event={"ID":"c3b6ff81-a886-4fe4-a1a1-4c4d43f6068e","Type":"ContainerStarted","Data":"0f0afb2b9871ffdd098f1a884c9d8d789e4e2c89d581c0bb579a7ae78ca76ef4"}
Sep 17 22:03:10.466119 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:10.465746    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-qttx6"
Sep 17 22:03:10.482737 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:10.482676    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-qttx6" podStartSLOduration=1.98116727 podStartE2EDuration="3.482660453s" totalImagesPullingTime="1.501493183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:10.481160672 +0000 UTC m=+289.564748644" watchObservedRunningTime="2026-09-17 22:03:10.482660453 +0000 UTC m=+289.566248401"
Sep 17 22:03:11.472686 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:11.472645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw" event={"ID":"3dc861b2-f144-410f-a853-3dc1a277f4b5","Type":"ContainerStarted","Data":"3b36d6575a042b358a1481e868db33b80816d1e1ffb5099fc59467c503b37c71"}
Sep 17 22:03:11.473085 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:11.472723    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:11.488740 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:11.488649    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw" podStartSLOduration=2.731550799 podStartE2EDuration="5.488630986s" totalImagesPullingTime="2.757080187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:03:11.487548825 +0000 UTC m=+290.571136805" watchObservedRunningTime="2026-09-17 22:03:11.488630986 +0000 UTC m=+290.572218962"
Sep 17 22:03:15.877943 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:15.877900    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:15.930067 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:15.930036    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:16.773610 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:16.773575    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:03:22.482633 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:22.482599    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-gtgkw"
Sep 17 22:03:23.071924 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:23.071848    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 22:03:23.072188 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:23.072167    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 22:03:23.078379 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:23.078359    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 22:03:23.078748 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:23.078727    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 22:03:23.084342 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:23.084323    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 22:03:26.443908 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:26.443877    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-djjcn"
Sep 17 22:03:31.475700 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:31.475662    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-qttx6"
Sep 17 22:03:58.663617 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:03:58.663582    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:00.110260 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:00.110220    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:23.102191 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.102155    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:23.168253 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.168220    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-79b679679b-htw6q"]
Sep 17 22:04:23.170992 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.170969    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:04:23.173345 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.173324    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 22:04:23.174243 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.174222    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 22:04:23.174243 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.174234    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-8m2jt\""
Sep 17 22:04:23.174423 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.174264    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 22:04:23.189069 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.189042    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-htw6q"]
Sep 17 22:04:23.189231 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.189075    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:23.262436 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.262394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtvgd\" (UniqueName: \"kubernetes.io/projected/8cd60064-4b1f-4a50-aff2-656326cb6465-kube-api-access-jtvgd\") pod \"kserve-controller-manager-79b679679b-htw6q\" (UID: \"8cd60064-4b1f-4a50-aff2-656326cb6465\") " pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:04:23.262623 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.262537    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cd60064-4b1f-4a50-aff2-656326cb6465-cert\") pod \"kserve-controller-manager-79b679679b-htw6q\" (UID: \"8cd60064-4b1f-4a50-aff2-656326cb6465\") " pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:04:23.316752 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.316715    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:23.363307 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.363224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtvgd\" (UniqueName: \"kubernetes.io/projected/8cd60064-4b1f-4a50-aff2-656326cb6465-kube-api-access-jtvgd\") pod \"kserve-controller-manager-79b679679b-htw6q\" (UID: \"8cd60064-4b1f-4a50-aff2-656326cb6465\") " pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:04:23.363465 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.363341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cd60064-4b1f-4a50-aff2-656326cb6465-cert\") pod \"kserve-controller-manager-79b679679b-htw6q\" (UID: \"8cd60064-4b1f-4a50-aff2-656326cb6465\") " pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:04:23.365792 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.365767    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cd60064-4b1f-4a50-aff2-656326cb6465-cert\") pod \"kserve-controller-manager-79b679679b-htw6q\" (UID: \"8cd60064-4b1f-4a50-aff2-656326cb6465\") " pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:04:23.374072 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.374047    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtvgd\" (UniqueName: \"kubernetes.io/projected/8cd60064-4b1f-4a50-aff2-656326cb6465-kube-api-access-jtvgd\") pod \"kserve-controller-manager-79b679679b-htw6q\" (UID: \"8cd60064-4b1f-4a50-aff2-656326cb6465\") " pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:04:23.409694 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.409661    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:23.482924 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.482874    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:04:23.754934 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.754910    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-htw6q"]
Sep 17 22:04:23.757490 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:04:23.757458    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd60064_4b1f_4a50_aff2_656326cb6465.slice/crio-bed2b11cedf0ed56d942185a9ae8fd593f9cccab2a1d30abece565b4cb6eeb11 WatchSource:0}: Error finding container bed2b11cedf0ed56d942185a9ae8fd593f9cccab2a1d30abece565b4cb6eeb11: Status 404 returned error can't find the container with id bed2b11cedf0ed56d942185a9ae8fd593f9cccab2a1d30abece565b4cb6eeb11
Sep 17 22:04:23.759273 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.759253    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:04:23.780835 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.780803    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:23.796812 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.796776    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-htw6q" event={"ID":"8cd60064-4b1f-4a50-aff2-656326cb6465","Type":"ContainerStarted","Data":"bed2b11cedf0ed56d942185a9ae8fd593f9cccab2a1d30abece565b4cb6eeb11"}
Sep 17 22:04:23.881204 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:23.881160    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:26.819054 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:26.818948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-htw6q" event={"ID":"8cd60064-4b1f-4a50-aff2-656326cb6465","Type":"ContainerStarted","Data":"e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e"}
Sep 17 22:04:26.819054 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:26.819051    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:04:26.836783 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:26.836720    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-79b679679b-htw6q" podStartSLOduration=1.184292514 podStartE2EDuration="3.836699766s" totalImagesPullingTime="2.652407252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:04:26.834345311 +0000 UTC m=+365.917933271" watchObservedRunningTime="2026-09-17 22:04:26.836699766 +0000 UTC m=+365.920287717"
Sep 17 22:04:54.559941 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.559902    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55f85586b-6mlp9"]
Sep 17 22:04:54.563429 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.563393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.566113 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.566089    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 22:04:54.567112 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.567090    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 22:04:54.567231 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.567101    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 22:04:54.567385 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.567368    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 22:04:54.567470 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.567448    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 22:04:54.567533 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.567470    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 22:04:54.567611 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.567596    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 22:04:54.568894 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.568872    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5m25g\""
Sep 17 22:04:54.572171 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.572126    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 22:04:54.574814 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.574791    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f85586b-6mlp9"]
Sep 17 22:04:54.611966 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.611934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-service-ca\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.612183 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.612003    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-console-oauth-config\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.612183 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.612030    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-oauth-serving-cert\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.612183 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.612051    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw48n\" (UniqueName: \"kubernetes.io/projected/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-kube-api-access-kw48n\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.612183 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.612082    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-console-config\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.612339 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.612214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-console-serving-cert\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.612339 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.612263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-trusted-ca-bundle\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.617079 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.617054    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:54.712934 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.712892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-console-oauth-config\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.712934 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.712931    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-oauth-serving-cert\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.712934 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.712945    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw48n\" (UniqueName: \"kubernetes.io/projected/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-kube-api-access-kw48n\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.713292 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.712971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-console-config\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.713292 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.713035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-console-serving-cert\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.713292 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.713061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-trusted-ca-bundle\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.713292 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.713100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-service-ca\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.713922 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.713894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-console-config\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.714027 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.713931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-service-ca\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.714027 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.713953    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-oauth-serving-cert\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.714112 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.714032    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-trusted-ca-bundle\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.715677 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.715651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-console-oauth-config\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.715677 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.715669    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-console-serving-cert\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.721039 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.721020    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw48n\" (UniqueName: \"kubernetes.io/projected/7a9337c2-28eb-4d24-9247-5dd1b9a838c4-kube-api-access-kw48n\") pod \"console-55f85586b-6mlp9\" (UID: \"7a9337c2-28eb-4d24-9247-5dd1b9a838c4\") " pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:54.760810 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.760772    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:54.868432 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.868352    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:54.874317 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:54.874285    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:04:55.155228 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:55.155200    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f85586b-6mlp9"]
Sep 17 22:04:55.157175 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:04:55.157147    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9337c2_28eb_4d24_9247_5dd1b9a838c4.slice/crio-28f5e0777ee440599da38a7bb0bd81a4f4f3b02eadca6a5b30909c9f7547ac3a WatchSource:0}: Error finding container 28f5e0777ee440599da38a7bb0bd81a4f4f3b02eadca6a5b30909c9f7547ac3a: Status 404 returned error can't find the container with id 28f5e0777ee440599da38a7bb0bd81a4f4f3b02eadca6a5b30909c9f7547ac3a
Sep 17 22:04:55.358702 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:55.358668    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:55.456301 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:55.456224    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:04:55.955164 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:55.955108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f85586b-6mlp9" event={"ID":"7a9337c2-28eb-4d24-9247-5dd1b9a838c4","Type":"ContainerStarted","Data":"c57b96aa2cc9f294f844c2291c831a557a2df6b08a75738229ce49d45a58adcc"}
Sep 17 22:04:55.955164 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:55.955172    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f85586b-6mlp9" event={"ID":"7a9337c2-28eb-4d24-9247-5dd1b9a838c4","Type":"ContainerStarted","Data":"28f5e0777ee440599da38a7bb0bd81a4f4f3b02eadca6a5b30909c9f7547ac3a"}
Sep 17 22:04:55.974791 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:55.974738    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55f85586b-6mlp9" podStartSLOduration=1.974720204 podStartE2EDuration="1.974720204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:04:55.972720778 +0000 UTC m=+395.056308742" watchObservedRunningTime="2026-09-17 22:04:55.974720204 +0000 UTC m=+395.058308153"
Sep 17 22:04:57.829157 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:04:57.829104    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:05:04.875341 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:04.875300    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:05:04.875813 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:04.875403    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:05:04.879946 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:04.879917    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:05:04.998310 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:04.998277    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55f85586b-6mlp9"
Sep 17 22:05:05.276012 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:05.275972    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:05.355500 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:05.355463    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:06.230591 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.230557    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-htw6q"]
Sep 17 22:05:06.231054 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.230768    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-79b679679b-htw6q" podUID="8cd60064-4b1f-4a50-aff2-656326cb6465" containerName="manager" containerID="cri-o://e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e" gracePeriod=10
Sep 17 22:05:06.256706 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.256676    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-79b679679b-4622b"]
Sep 17 22:05:06.258828 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.258812    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-4622b"
Sep 17 22:05:06.268943 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.268914    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-4622b"]
Sep 17 22:05:06.306421 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.306396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwljz\" (UniqueName: \"kubernetes.io/projected/4d5b680f-dc39-4d5b-9ad2-85890dd122fe-kube-api-access-hwljz\") pod \"kserve-controller-manager-79b679679b-4622b\" (UID: \"4d5b680f-dc39-4d5b-9ad2-85890dd122fe\") " pod="kserve/kserve-controller-manager-79b679679b-4622b"
Sep 17 22:05:06.306540 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.306439    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d5b680f-dc39-4d5b-9ad2-85890dd122fe-cert\") pod \"kserve-controller-manager-79b679679b-4622b\" (UID: \"4d5b680f-dc39-4d5b-9ad2-85890dd122fe\") " pod="kserve/kserve-controller-manager-79b679679b-4622b"
Sep 17 22:05:06.407764 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.407733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwljz\" (UniqueName: \"kubernetes.io/projected/4d5b680f-dc39-4d5b-9ad2-85890dd122fe-kube-api-access-hwljz\") pod \"kserve-controller-manager-79b679679b-4622b\" (UID: \"4d5b680f-dc39-4d5b-9ad2-85890dd122fe\") " pod="kserve/kserve-controller-manager-79b679679b-4622b"
Sep 17 22:05:06.407926 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.407805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d5b680f-dc39-4d5b-9ad2-85890dd122fe-cert\") pod \"kserve-controller-manager-79b679679b-4622b\" (UID: \"4d5b680f-dc39-4d5b-9ad2-85890dd122fe\") " pod="kserve/kserve-controller-manager-79b679679b-4622b"
Sep 17 22:05:06.410554 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.410519    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d5b680f-dc39-4d5b-9ad2-85890dd122fe-cert\") pod \"kserve-controller-manager-79b679679b-4622b\" (UID: \"4d5b680f-dc39-4d5b-9ad2-85890dd122fe\") " pod="kserve/kserve-controller-manager-79b679679b-4622b"
Sep 17 22:05:06.417558 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.417533    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwljz\" (UniqueName: \"kubernetes.io/projected/4d5b680f-dc39-4d5b-9ad2-85890dd122fe-kube-api-access-hwljz\") pod \"kserve-controller-manager-79b679679b-4622b\" (UID: \"4d5b680f-dc39-4d5b-9ad2-85890dd122fe\") " pod="kserve/kserve-controller-manager-79b679679b-4622b"
Sep 17 22:05:06.476061 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.476034    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:05:06.508537 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.508457    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtvgd\" (UniqueName: \"kubernetes.io/projected/8cd60064-4b1f-4a50-aff2-656326cb6465-kube-api-access-jtvgd\") pod \"8cd60064-4b1f-4a50-aff2-656326cb6465\" (UID: \"8cd60064-4b1f-4a50-aff2-656326cb6465\") "
Sep 17 22:05:06.508537 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.508489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cd60064-4b1f-4a50-aff2-656326cb6465-cert\") pod \"8cd60064-4b1f-4a50-aff2-656326cb6465\" (UID: \"8cd60064-4b1f-4a50-aff2-656326cb6465\") "
Sep 17 22:05:06.510648 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.510615    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd60064-4b1f-4a50-aff2-656326cb6465-kube-api-access-jtvgd" (OuterVolumeSpecName: "kube-api-access-jtvgd") pod "8cd60064-4b1f-4a50-aff2-656326cb6465" (UID: "8cd60064-4b1f-4a50-aff2-656326cb6465"). InnerVolumeSpecName "kube-api-access-jtvgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:05:06.510780 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.510630    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd60064-4b1f-4a50-aff2-656326cb6465-cert" (OuterVolumeSpecName: "cert") pod "8cd60064-4b1f-4a50-aff2-656326cb6465" (UID: "8cd60064-4b1f-4a50-aff2-656326cb6465"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:05:06.605056 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.605013    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-4622b"
Sep 17 22:05:06.609254 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.609232    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtvgd\" (UniqueName: \"kubernetes.io/projected/8cd60064-4b1f-4a50-aff2-656326cb6465-kube-api-access-jtvgd\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:05:06.609346 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.609258    2583 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8cd60064-4b1f-4a50-aff2-656326cb6465-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:05:06.881856 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:06.881822    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-4622b"]
Sep 17 22:05:06.884536 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:05:06.884498    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5b680f_dc39_4d5b_9ad2_85890dd122fe.slice/crio-9abd83ea928e487ef36c57e86f42a191205e800c4f8d142023de518c6484f31f WatchSource:0}: Error finding container 9abd83ea928e487ef36c57e86f42a191205e800c4f8d142023de518c6484f31f: Status 404 returned error can't find the container with id 9abd83ea928e487ef36c57e86f42a191205e800c4f8d142023de518c6484f31f
Sep 17 22:05:07.004057 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:07.004026    2583 generic.go:356] "Generic (PLEG): container finished" podID="8cd60064-4b1f-4a50-aff2-656326cb6465" containerID="e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e" exitCode=0
Sep 17 22:05:07.004262 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:07.004072    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-79b679679b-htw6q"
Sep 17 22:05:07.004262 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:07.004097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-htw6q" event={"ID":"8cd60064-4b1f-4a50-aff2-656326cb6465","Type":"ContainerDied","Data":"e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e"}
Sep 17 22:05:07.004262 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:07.004178    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-htw6q" event={"ID":"8cd60064-4b1f-4a50-aff2-656326cb6465","Type":"ContainerDied","Data":"bed2b11cedf0ed56d942185a9ae8fd593f9cccab2a1d30abece565b4cb6eeb11"}
Sep 17 22:05:07.004262 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:07.004214    2583 scope.go:117] "RemoveContainer" containerID="e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e"
Sep 17 22:05:07.005465 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:07.005442    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-4622b" event={"ID":"4d5b680f-dc39-4d5b-9ad2-85890dd122fe","Type":"ContainerStarted","Data":"9abd83ea928e487ef36c57e86f42a191205e800c4f8d142023de518c6484f31f"}
Sep 17 22:05:07.012618 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:07.012598    2583 scope.go:117] "RemoveContainer" containerID="e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e"
Sep 17 22:05:07.012900 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:05:07.012883    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e\": container with ID starting with e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e not found: ID does not exist" containerID="e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e"
Sep 17 22:05:07.012939 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:07.012910    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e"} err="failed to get container status \"e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e\": rpc error: code = NotFound desc = could not find container \"e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e\": container with ID starting with e4d9dd3d316cc6e1a7f1eda2d653c1ca1714d51396d9beea435805d5e111f39e not found: ID does not exist"
Sep 17 22:05:07.027238 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:07.027212    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-htw6q"]
Sep 17 22:05:07.029723 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:07.029701    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-79b679679b-htw6q"]
Sep 17 22:05:07.186020 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:07.185941    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cd60064-4b1f-4a50-aff2-656326cb6465" path="/var/lib/kubelet/pods/8cd60064-4b1f-4a50-aff2-656326cb6465/volumes"
Sep 17 22:05:08.012086 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:08.012046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-79b679679b-4622b" event={"ID":"4d5b680f-dc39-4d5b-9ad2-85890dd122fe","Type":"ContainerStarted","Data":"3ca5ea68d9e585d914c740cf4aae95fda66cc5f20b9211deb3f29b93b9917639"}
Sep 17 22:05:08.012476 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:08.012182    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-79b679679b-4622b"
Sep 17 22:05:08.029064 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:08.029012    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-79b679679b-4622b" podStartSLOduration=1.503494757 podStartE2EDuration="2.028995056s" totalImagesPullingTime="525.500299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:05:08.028292751 +0000 UTC m=+407.111880785" watchObservedRunningTime="2026-09-17 22:05:08.028995056 +0000 UTC m=+407.112583006"
Sep 17 22:05:09.082997 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:09.082960    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:09.179925 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:09.179893    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:10.831819 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:10.831783    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:10.893512 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:10.893466    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:19.987218 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:19.987182    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:39.022805 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:39.022717    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-79b679679b-4622b"
Sep 17 22:05:40.274508 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:40.274468    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:40.357053 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:40.357020    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:40.512044 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:40.512006    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:40.612942 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:40.612861    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:40.712991 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:40.712955    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:41.471245 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:41.471211    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:05:41.569892 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:05:41.569856    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:06:33.054670 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:06:33.054633    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:15.225989 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:15.225909    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:15.325895 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:15.325856    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:26.680147 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:26.680101    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:26.835315 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:26.835274    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:26.944492 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:26.944383    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:27.028231 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.028199    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"]
Sep 17 22:07:27.028584 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.028571    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd60064-4b1f-4a50-aff2-656326cb6465" containerName="manager"
Sep 17 22:07:27.028638 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.028586    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd60064-4b1f-4a50-aff2-656326cb6465" containerName="manager"
Sep 17 22:07:27.028692 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.028642    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd60064-4b1f-4a50-aff2-656326cb6465" containerName="manager"
Sep 17 22:07:27.031286 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.031266    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.036978 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.036958    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-d91b5-predictor-serving-cert\""
Sep 17 22:07:27.037264 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.037244    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-d91b5-kube-rbac-proxy-sar-config\""
Sep 17 22:07:27.037378 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.037304    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:07:27.037448 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.037400    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:07:27.037558 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.037539    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:07:27.037911 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.037541    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:07:27.038388 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.038119    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:07:27.051803 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.051773    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"]
Sep 17 22:07:27.073367 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.073337    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:27.155510 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.155474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:27.215990 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.215905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/980303d1-9b9e-4d07-91eb-769ac8f69542-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.215990 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.215946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/980303d1-9b9e-4d07-91eb-769ac8f69542-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.216261 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.216026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.216261 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.216108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqxg\" (UniqueName: \"kubernetes.io/projected/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-api-access-cxqxg\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.216261 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.216164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.255691 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.255663    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:07:27.316600 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.316568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/980303d1-9b9e-4d07-91eb-769ac8f69542-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.316600 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.316606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/980303d1-9b9e-4d07-91eb-769ac8f69542-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.316857 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.316656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.316857 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.316711    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqxg\" (UniqueName: \"kubernetes.io/projected/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-api-access-cxqxg\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.316857 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.316737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.317097 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.317069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/980303d1-9b9e-4d07-91eb-769ac8f69542-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.317453 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.317431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.317544 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.317476    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.319178 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.319158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/980303d1-9b9e-4d07-91eb-769ac8f69542-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.324382 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.324361    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqxg\" (UniqueName: \"kubernetes.io/projected/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-api-access-cxqxg\") pod \"isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.342881 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.342844    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:27.621077 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:27.621047    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"]
Sep 17 22:07:27.623601 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:07:27.623565    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980303d1_9b9e_4d07_91eb_769ac8f69542.slice/crio-8854f4f389d7133a75b7976dd03eb3a87c516a458a54cb86f7fca2345e661b9d WatchSource:0}: Error finding container 8854f4f389d7133a75b7976dd03eb3a87c516a458a54cb86f7fca2345e661b9d: Status 404 returned error can't find the container with id 8854f4f389d7133a75b7976dd03eb3a87c516a458a54cb86f7fca2345e661b9d
Sep 17 22:07:28.633476 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:28.633435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" event={"ID":"980303d1-9b9e-4d07-91eb-769ac8f69542","Type":"ContainerStarted","Data":"8854f4f389d7133a75b7976dd03eb3a87c516a458a54cb86f7fca2345e661b9d"}
Sep 17 22:07:31.651525 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:31.651489    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" event={"ID":"980303d1-9b9e-4d07-91eb-769ac8f69542","Type":"ContainerStarted","Data":"d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc"}
Sep 17 22:07:34.667119 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:34.667084    2583 generic.go:356] "Generic (PLEG): container finished" podID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerID="d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc" exitCode=0
Sep 17 22:07:34.667492 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:34.667159    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" event={"ID":"980303d1-9b9e-4d07-91eb-769ac8f69542","Type":"ContainerDied","Data":"d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc"}
Sep 17 22:07:47.745426 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:47.745385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" event={"ID":"980303d1-9b9e-4d07-91eb-769ac8f69542","Type":"ContainerStarted","Data":"ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4"}
Sep 17 22:07:50.762497 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:50.762458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" event={"ID":"980303d1-9b9e-4d07-91eb-769ac8f69542","Type":"ContainerStarted","Data":"93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2"}
Sep 17 22:07:52.775529 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:52.775488    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" event={"ID":"980303d1-9b9e-4d07-91eb-769ac8f69542","Type":"ContainerStarted","Data":"74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4"}
Sep 17 22:07:52.775985 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:52.775691    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:52.775985 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:52.775864    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:52.777268 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:52.777239    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:07:52.797469 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:52.797411    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podStartSLOduration=1.74928588 podStartE2EDuration="25.797394264s" totalImagesPullingTime="20.295298858s" totalInitContainerRuntime="3.752809526s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:07:27 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:07:52.795110201 +0000 UTC m=+571.878698177" watchObservedRunningTime="2026-09-17 22:07:52.797394264 +0000 UTC m=+571.880982280"
Sep 17 22:07:53.780116 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:53.780079    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:53.780540 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:53.780326    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:07:53.781536 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:53.781509    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:07:54.784345 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:54.784303    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:07:54.784822 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:54.784744    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:07:54.788123 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:54.788103    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:07:55.788639 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:55.788600    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:07:55.789092 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:07:55.788936    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:08:05.789372 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:05.789319    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:08:05.789855 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:05.789828    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:08:12.853502 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:12.853456    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:08:15.788592 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:15.788543    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:08:15.789044 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:15.789003    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:08:23.101433 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:23.101400    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 22:08:23.103890 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:23.103864    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 22:08:23.106924 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:23.106876    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 22:08:23.109720 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:23.109701    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 22:08:25.788932 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:25.788886    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:08:25.789361 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:25.789333    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:08:35.789370 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:35.789263    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:08:35.790725 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:35.789707    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:08:45.789082 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:45.789034    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:08:45.789534 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:45.789458    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:08:55.789295 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:55.789239    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:08:55.789829 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:08:55.789648    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:09:00.180866 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:00.180822    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:09:00.181303 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:00.181162    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:09:10.181324 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:10.181286    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:09:10.181763 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:10.181375    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:09:14.070317 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:14.070283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:15.270531 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:15.270492    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:16.663982 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:16.663940    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:21.694818 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:21.694783    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"]
Sep 17 22:09:21.702478 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:21.695511    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" containerID="cri-o://ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4" gracePeriod=30
Sep 17 22:09:21.702478 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:21.695735    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kube-rbac-proxy" containerID="cri-o://93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2" gracePeriod=30
Sep 17 22:09:21.702478 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:21.695710    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" containerID="cri-o://74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4" gracePeriod=30
Sep 17 22:09:21.710476 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:21.710445    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:21.809170 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:21.809117    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:21.934039 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:21.933994    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:22.003817 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:22.003740    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:22.187813 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:22.187780    2583 generic.go:356] "Generic (PLEG): container finished" podID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerID="93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2" exitCode=2
Sep 17 22:09:22.187981 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:22.187826    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" event={"ID":"980303d1-9b9e-4d07-91eb-769ac8f69542","Type":"ContainerDied","Data":"93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2"}
Sep 17 22:09:22.340594 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:22.340513    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:09:24.784478 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:24.784436    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 22:09:27.211421 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:27.211389    2583 generic.go:356] "Generic (PLEG): container finished" podID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerID="ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4" exitCode=0
Sep 17 22:09:27.211859 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:27.211462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" event={"ID":"980303d1-9b9e-4d07-91eb-769ac8f69542","Type":"ContainerDied","Data":"ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4"}
Sep 17 22:09:29.784833 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:29.784785    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 22:09:30.180846 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:30.180802    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:09:30.181216 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:30.181194    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:09:34.785295 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:34.785254    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 22:09:34.785694 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:34.785400    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:09:39.784992 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:39.784939    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 22:09:40.180921 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:40.180875    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:09:40.181270 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:40.181241    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:09:44.784729 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:44.784680    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 22:09:49.784533 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:49.784488    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 17 22:09:50.180820 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:50.180772    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 17 22:09:50.181027 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:50.180897    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:09:50.181184 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:50.181123    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:09:50.181274 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:50.181257    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:09:51.858934 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:51.858908    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:09:51.900230 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:51.900195    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxqxg\" (UniqueName: \"kubernetes.io/projected/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-api-access-cxqxg\") pod \"980303d1-9b9e-4d07-91eb-769ac8f69542\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") "
Sep 17 22:09:51.900405 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:51.900251    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/980303d1-9b9e-4d07-91eb-769ac8f69542-proxy-tls\") pod \"980303d1-9b9e-4d07-91eb-769ac8f69542\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") "
Sep 17 22:09:51.900405 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:51.900318    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-cabundle-cert\") pod \"980303d1-9b9e-4d07-91eb-769ac8f69542\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") "
Sep 17 22:09:51.900405 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:51.900362    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/980303d1-9b9e-4d07-91eb-769ac8f69542-kserve-provision-location\") pod \"980303d1-9b9e-4d07-91eb-769ac8f69542\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") "
Sep 17 22:09:51.900574 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:51.900417    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-rbac-proxy-sar-config\") pod \"980303d1-9b9e-4d07-91eb-769ac8f69542\" (UID: \"980303d1-9b9e-4d07-91eb-769ac8f69542\") "
Sep 17 22:09:51.900796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:51.900754    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/980303d1-9b9e-4d07-91eb-769ac8f69542-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "980303d1-9b9e-4d07-91eb-769ac8f69542" (UID: "980303d1-9b9e-4d07-91eb-769ac8f69542"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:09:51.900796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:51.900776    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "980303d1-9b9e-4d07-91eb-769ac8f69542" (UID: "980303d1-9b9e-4d07-91eb-769ac8f69542"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:51.900970 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:51.900899    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "980303d1-9b9e-4d07-91eb-769ac8f69542" (UID: "980303d1-9b9e-4d07-91eb-769ac8f69542"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:09:51.902551 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:51.902525    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980303d1-9b9e-4d07-91eb-769ac8f69542-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "980303d1-9b9e-4d07-91eb-769ac8f69542" (UID: "980303d1-9b9e-4d07-91eb-769ac8f69542"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:09:51.902641 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:51.902610    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-api-access-cxqxg" (OuterVolumeSpecName: "kube-api-access-cxqxg") pod "980303d1-9b9e-4d07-91eb-769ac8f69542" (UID: "980303d1-9b9e-4d07-91eb-769ac8f69542"). InnerVolumeSpecName "kube-api-access-cxqxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:09:52.001308 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.001204    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-cabundle-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:09:52.001308 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.001246    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/980303d1-9b9e-4d07-91eb-769ac8f69542-kserve-provision-location\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:09:52.001308 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.001255    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:09:52.001308 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.001264    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxqxg\" (UniqueName: \"kubernetes.io/projected/980303d1-9b9e-4d07-91eb-769ac8f69542-kube-api-access-cxqxg\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:09:52.001308 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.001273    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/980303d1-9b9e-4d07-91eb-769ac8f69542-proxy-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:09:52.326501 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.326413    2583 generic.go:356] "Generic (PLEG): container finished" podID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerID="74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4" exitCode=0
Sep 17 22:09:52.326501 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.326481    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"
Sep 17 22:09:52.326501 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.326491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" event={"ID":"980303d1-9b9e-4d07-91eb-769ac8f69542","Type":"ContainerDied","Data":"74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4"}
Sep 17 22:09:52.326720 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.326571    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l" event={"ID":"980303d1-9b9e-4d07-91eb-769ac8f69542","Type":"ContainerDied","Data":"8854f4f389d7133a75b7976dd03eb3a87c516a458a54cb86f7fca2345e661b9d"}
Sep 17 22:09:52.326720 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.326602    2583 scope.go:117] "RemoveContainer" containerID="74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4"
Sep 17 22:09:52.336439 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.336411    2583 scope.go:117] "RemoveContainer" containerID="93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2"
Sep 17 22:09:52.344726 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.344707    2583 scope.go:117] "RemoveContainer" containerID="ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4"
Sep 17 22:09:52.350386 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.350363    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"]
Sep 17 22:09:52.353619 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.353595    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d91b5-predictor-74bbffb4f-6ln5l"]
Sep 17 22:09:52.353888 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.353869    2583 scope.go:117] "RemoveContainer" containerID="d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc"
Sep 17 22:09:52.362011 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.361993    2583 scope.go:117] "RemoveContainer" containerID="74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4"
Sep 17 22:09:52.362376 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:09:52.362357    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4\": container with ID starting with 74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4 not found: ID does not exist" containerID="74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4"
Sep 17 22:09:52.362427 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.362387    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4"} err="failed to get container status \"74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4\": rpc error: code = NotFound desc = could not find container \"74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4\": container with ID starting with 74da23e4a1bec13b40062347abf93fa458767cbc4eb23c9e1138f6d1cce3bfe4 not found: ID does not exist"
Sep 17 22:09:52.362427 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.362406    2583 scope.go:117] "RemoveContainer" containerID="93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2"
Sep 17 22:09:52.362705 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:09:52.362683    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2\": container with ID starting with 93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2 not found: ID does not exist" containerID="93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2"
Sep 17 22:09:52.362754 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.362711    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2"} err="failed to get container status \"93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2\": rpc error: code = NotFound desc = could not find container \"93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2\": container with ID starting with 93a3ef9448c72e0215ec0b5f28f940f1c957500c3cd8a2fd41821a5bb31289a2 not found: ID does not exist"
Sep 17 22:09:52.362754 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.362728    2583 scope.go:117] "RemoveContainer" containerID="ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4"
Sep 17 22:09:52.362979 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:09:52.362960    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4\": container with ID starting with ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4 not found: ID does not exist" containerID="ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4"
Sep 17 22:09:52.363045 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.362988    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4"} err="failed to get container status \"ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4\": rpc error: code = NotFound desc = could not find container \"ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4\": container with ID starting with ac80dd57e6c7189bcff4db8f5291870d51d2b0bbb2ad8873eea6e699235e6cd4 not found: ID does not exist"
Sep 17 22:09:52.363045 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.363010    2583 scope.go:117] "RemoveContainer" containerID="d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc"
Sep 17 22:09:52.363287 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:09:52.363267    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc\": container with ID starting with d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc not found: ID does not exist" containerID="d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc"
Sep 17 22:09:52.363340 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:52.363292    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc"} err="failed to get container status \"d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc\": rpc error: code = NotFound desc = could not find container \"d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc\": container with ID starting with d57e65511a6eafd329931bf36482bd04e081fe6aa57d77e6199c304feb767efc not found: ID does not exist"
Sep 17 22:09:53.186192 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:09:53.186158    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" path="/var/lib/kubelet/pods/980303d1-9b9e-4d07-91eb-769ac8f69542/volumes"
Sep 17 22:10:46.818028 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:46.817989    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:10:51.971729 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:51.971696    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:10:52.039060 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039026    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"]
Sep 17 22:10:52.039485 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039465    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="storage-initializer"
Sep 17 22:10:52.039646 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039628    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="storage-initializer"
Sep 17 22:10:52.039728 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039692    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kube-rbac-proxy"
Sep 17 22:10:52.039728 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039703    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kube-rbac-proxy"
Sep 17 22:10:52.039728 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039728    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container"
Sep 17 22:10:52.039728 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039736    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container"
Sep 17 22:10:52.039933 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039759    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent"
Sep 17 22:10:52.039933 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039767    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent"
Sep 17 22:10:52.039933 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039890    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kube-rbac-proxy"
Sep 17 22:10:52.039933 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039907    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="agent"
Sep 17 22:10:52.039933 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.039916    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="980303d1-9b9e-4d07-91eb-769ac8f69542" containerName="kserve-container"
Sep 17 22:10:52.043303 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.043284    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.048695 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.048672    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:10:52.049893 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.049723    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 22:10:52.050038 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.050021    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 22:10:52.050115 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.050091    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:10:52.050597 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.050581    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:10:52.050998 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.050980    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:10:52.051567 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.051550    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:10:52.061332 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.061308    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:10:52.079242 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.079217    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"]
Sep 17 22:10:52.128267 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.128237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpzg7\" (UniqueName: \"kubernetes.io/projected/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-api-access-gpzg7\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.128397 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.128276    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.128397 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.128301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.128397 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.128383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.128520 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.128500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.229824 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.229738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.229824 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.229803    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpzg7\" (UniqueName: \"kubernetes.io/projected/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-api-access-gpzg7\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.230051 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.229847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.230051 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.229876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.230051 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.229905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.230051 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:10:52.230048    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 22:10:52.230286 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:10:52.230120    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-proxy-tls podName:dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d nodeName:}" failed. No retries permitted until 2026-09-17 22:10:52.730099304 +0000 UTC m=+751.813687248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-proxy-tls") pod "isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" (UID: "dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 22:10:52.230286 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.230247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.230667 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.230648    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.230831 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.230799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.237931 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.237900    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpzg7\" (UniqueName: \"kubernetes.io/projected/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-api-access-gpzg7\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.455896 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.455857    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:10:52.521712 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.521645    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:10:52.734578 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.734529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.736924 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.736894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:52.953635 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:52.953600    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:53.226128 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:53.226101    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"]
Sep 17 22:10:53.228204 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:10:53.228170    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbd29f0_b8b4_4b8b_90a0_f0d524291c5d.slice/crio-b1319a6c5cb17a60090aca52e687bc675152155eb3a7c30d6d197bcfa62c4128 WatchSource:0}: Error finding container b1319a6c5cb17a60090aca52e687bc675152155eb3a7c30d6d197bcfa62c4128: Status 404 returned error can't find the container with id b1319a6c5cb17a60090aca52e687bc675152155eb3a7c30d6d197bcfa62c4128
Sep 17 22:10:53.593178 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:53.593068    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" event={"ID":"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d","Type":"ContainerStarted","Data":"dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4"}
Sep 17 22:10:53.593178 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:53.593130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" event={"ID":"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d","Type":"ContainerStarted","Data":"b1319a6c5cb17a60090aca52e687bc675152155eb3a7c30d6d197bcfa62c4128"}
Sep 17 22:10:57.611117 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:57.611085    2583 generic.go:356] "Generic (PLEG): container finished" podID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerID="dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4" exitCode=0
Sep 17 22:10:57.611521 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:57.611123    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" event={"ID":"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d","Type":"ContainerDied","Data":"dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4"}
Sep 17 22:10:58.389586 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:58.389544    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:10:58.618352 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:58.618318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" event={"ID":"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d","Type":"ContainerStarted","Data":"4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549"}
Sep 17 22:10:58.618769 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:58.618378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" event={"ID":"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d","Type":"ContainerStarted","Data":"c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880"}
Sep 17 22:10:58.618769 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:58.618657    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:58.618857 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:58.618800    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:10:58.620144 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:58.620111    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:10:58.638341 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:58.638298    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podStartSLOduration=2.923314145 podStartE2EDuration="6.638283122s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714968977s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:10:58.636325303 +0000 UTC m=+757.719913288" watchObservedRunningTime="2026-09-17 22:10:58.638283122 +0000 UTC m=+757.721871115"
Sep 17 22:10:59.624656 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:10:59.624615    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:11:01.849998 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:01.849965    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"]
Sep 17 22:11:01.853473 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:01.853452    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:01.854073 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:01.854032    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:11:01.855691 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:01.855673    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-49e3c-kube-rbac-proxy-sar-config\""
Sep 17 22:11:01.855795 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:01.855760    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-49e3c-serving-cert\""
Sep 17 22:11:01.861437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:01.861415    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"]
Sep 17 22:11:01.921075 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:01.921042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-proxy-tls\") pod \"model-chainer-raw-49e3c-6c8dd77d7b-jr2k5\" (UID: \"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f\") " pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:01.921277 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:01.921108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-openshift-service-ca-bundle\") pod \"model-chainer-raw-49e3c-6c8dd77d7b-jr2k5\" (UID: \"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f\") " pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:02.021838 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:02.021802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-openshift-service-ca-bundle\") pod \"model-chainer-raw-49e3c-6c8dd77d7b-jr2k5\" (UID: \"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f\") " pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:02.022032 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:02.021901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-proxy-tls\") pod \"model-chainer-raw-49e3c-6c8dd77d7b-jr2k5\" (UID: \"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f\") " pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:02.022032 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:11:02.021999    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-49e3c-serving-cert: secret "model-chainer-raw-49e3c-serving-cert" not found
Sep 17 22:11:02.022171 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:11:02.022062    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-proxy-tls podName:af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f nodeName:}" failed. No retries permitted until 2026-09-17 22:11:02.522044922 +0000 UTC m=+761.605632849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-proxy-tls") pod "model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" (UID: "af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f") : secret "model-chainer-raw-49e3c-serving-cert" not found
Sep 17 22:11:02.023059 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:02.023023    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-openshift-service-ca-bundle\") pod \"model-chainer-raw-49e3c-6c8dd77d7b-jr2k5\" (UID: \"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f\") " pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:02.344619 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:02.344577    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:11:02.526172 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:02.526118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-proxy-tls\") pod \"model-chainer-raw-49e3c-6c8dd77d7b-jr2k5\" (UID: \"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f\") " pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:02.528511 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:02.528488    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-proxy-tls\") pod \"model-chainer-raw-49e3c-6c8dd77d7b-jr2k5\" (UID: \"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f\") " pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:02.766286 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:02.766247    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:03.035681 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:03.035518    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"]
Sep 17 22:11:03.038533 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:11:03.038498    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5e24a1_3dac_4ffb_9b8f_ddc0bde91c1f.slice/crio-16cf0fc3d8d84c97f6dce7819461625ce285f8de3383c60491be7ba166edc3cd WatchSource:0}: Error finding container 16cf0fc3d8d84c97f6dce7819461625ce285f8de3383c60491be7ba166edc3cd: Status 404 returned error can't find the container with id 16cf0fc3d8d84c97f6dce7819461625ce285f8de3383c60491be7ba166edc3cd
Sep 17 22:11:03.040529 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:03.040510    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 22:11:03.643215 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:03.643175    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" event={"ID":"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f","Type":"ContainerStarted","Data":"16cf0fc3d8d84c97f6dce7819461625ce285f8de3383c60491be7ba166edc3cd"}
Sep 17 22:11:04.629894 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:04.629861    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:11:04.630400 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:04.630371    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:11:05.655866 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:05.655834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" event={"ID":"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f","Type":"ContainerStarted","Data":"938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa"}
Sep 17 22:11:05.656293 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:05.655980    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:05.671764 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:05.671717    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" podStartSLOduration=2.520993033 podStartE2EDuration="4.671703895s" totalImagesPullingTime="2.150710862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:11:05.670566317 +0000 UTC m=+764.754154265" watchObservedRunningTime="2026-09-17 22:11:05.671703895 +0000 UTC m=+764.755291844"
Sep 17 22:11:11.665658 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:11.665624    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:11.923814 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:11.923736    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"]
Sep 17 22:11:11.923957 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:11.923918    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" podUID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" containerName="model-chainer-raw-49e3c" containerID="cri-o://938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa" gracePeriod=30
Sep 17 22:11:11.944560 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:11.944532    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:11:12.067775 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.067738    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:11:12.170238 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.170205    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:11:12.183228 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.183162    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"]
Sep 17 22:11:12.186704 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.186675    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.189472 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.189452    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-5ac84-kube-rbac-proxy-sar-config\""
Sep 17 22:11:12.189633 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.189555    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-5ac84-predictor-serving-cert\""
Sep 17 22:11:12.197844 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.197814    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"]
Sep 17 22:11:12.267391 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.267355    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:11:12.314308 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.314260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.314308 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.314318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.314590 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.314349    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtdx8\" (UniqueName: \"kubernetes.io/projected/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-api-access-rtdx8\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.314590 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.314374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.314590 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.314405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.336755 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.336722    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:11:12.415219 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.415180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.415219 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.415223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.415451 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.415243    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtdx8\" (UniqueName: \"kubernetes.io/projected/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-api-access-rtdx8\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.415451 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.415275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.415451 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.415406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.415824 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.415793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.416015 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.415998    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.416066 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.416029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.417993 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.417970    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.423800 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.423779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtdx8\" (UniqueName: \"kubernetes.io/projected/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-api-access-rtdx8\") pod \"isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.498038 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.497956    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:12.670570 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.670524    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:11:12.779163 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:12.779112    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"]
Sep 17 22:11:12.783753 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:11:12.781188    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32d9318_556e_4c6d_b82f_bf5a83fd0d0f.slice/crio-1b684b5567e699a9339176869494b4394a8add356c9f67cfe9b8d26d34fc0785 WatchSource:0}: Error finding container 1b684b5567e699a9339176869494b4394a8add356c9f67cfe9b8d26d34fc0785: Status 404 returned error can't find the container with id 1b684b5567e699a9339176869494b4394a8add356c9f67cfe9b8d26d34fc0785
Sep 17 22:11:13.693356 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:13.693320    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" event={"ID":"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f","Type":"ContainerStarted","Data":"4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c"}
Sep 17 22:11:13.693840 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:13.693376    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" event={"ID":"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f","Type":"ContainerStarted","Data":"1b684b5567e699a9339176869494b4394a8add356c9f67cfe9b8d26d34fc0785"}
Sep 17 22:11:14.631235 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:14.631185    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:11:16.664253 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:16.664209    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" podUID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" containerName="model-chainer-raw-49e3c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:11:16.709555 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:16.709520    2583 generic.go:356] "Generic (PLEG): container finished" podID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerID="4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c" exitCode=0
Sep 17 22:11:16.709748 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:16.709599    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" event={"ID":"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f","Type":"ContainerDied","Data":"4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c"}
Sep 17 22:11:17.716916 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:17.716875    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" event={"ID":"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f","Type":"ContainerStarted","Data":"ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074"}
Sep 17 22:11:17.717312 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:17.716961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" event={"ID":"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f","Type":"ContainerStarted","Data":"629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0"}
Sep 17 22:11:17.717312 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:17.717276    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:17.735196 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:17.735119    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podStartSLOduration=2.20018159 podStartE2EDuration="5.735102611s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.534921021s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:11:17.734201837 +0000 UTC m=+776.817789807" watchObservedRunningTime="2026-09-17 22:11:17.735102611 +0000 UTC m=+776.818690542"
Sep 17 22:11:18.721945 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:18.721907    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:18.723399 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:18.723368    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:11:19.726238 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:19.726201    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:11:21.664649 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:21.664602    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" podUID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" containerName="model-chainer-raw-49e3c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:11:24.630806 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:24.630764    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:11:24.730925 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:24.730895    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:11:24.731361 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:24.731329    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:11:26.664657 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:26.664613    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" podUID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" containerName="model-chainer-raw-49e3c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:11:26.665097 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:26.664731    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:31.664231 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:31.664191    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" podUID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" containerName="model-chainer-raw-49e3c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:11:34.630616 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:34.630531    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:11:34.731353 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:34.731317    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:11:36.664648 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:36.664613    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" podUID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" containerName="model-chainer-raw-49e3c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:11:41.663941 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:41.663897    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" podUID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" containerName="model-chainer-raw-49e3c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:11:42.069553 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.069523    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:42.181799 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.181707    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-proxy-tls\") pod \"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f\" (UID: \"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f\") "
Sep 17 22:11:42.181799 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.181750    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-openshift-service-ca-bundle\") pod \"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f\" (UID: \"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f\") "
Sep 17 22:11:42.182128 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.182100    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" (UID: "af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:11:42.183797 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.183763    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" (UID: "af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:11:42.282419 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.282382    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-proxy-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:11:42.282419 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.282419    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f-openshift-service-ca-bundle\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:11:42.836304 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.836268    2583 generic.go:356] "Generic (PLEG): container finished" podID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" containerID="938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa" exitCode=0
Sep 17 22:11:42.837131 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.836308    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"
Sep 17 22:11:42.837131 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.836359    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" event={"ID":"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f","Type":"ContainerDied","Data":"938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa"}
Sep 17 22:11:42.837131 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.836425    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5" event={"ID":"af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f","Type":"ContainerDied","Data":"16cf0fc3d8d84c97f6dce7819461625ce285f8de3383c60491be7ba166edc3cd"}
Sep 17 22:11:42.837131 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.836463    2583 scope.go:117] "RemoveContainer" containerID="938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa"
Sep 17 22:11:42.845222 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.845204    2583 scope.go:117] "RemoveContainer" containerID="938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa"
Sep 17 22:11:42.845480 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:11:42.845460    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa\": container with ID starting with 938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa not found: ID does not exist" containerID="938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa"
Sep 17 22:11:42.845526 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.845490    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa"} err="failed to get container status \"938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa\": rpc error: code = NotFound desc = could not find container \"938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa\": container with ID starting with 938003517b9508a6015e226ef7228a084aa9fce2ec511867e54a25356e87d5fa not found: ID does not exist"
Sep 17 22:11:42.858797 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.858773    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"]
Sep 17 22:11:42.862071 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:42.862050    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-49e3c-6c8dd77d7b-jr2k5"]
Sep 17 22:11:43.186896 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:43.186862    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" path="/var/lib/kubelet/pods/af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f/volumes"
Sep 17 22:11:44.630577 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:44.630529    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:11:44.731455 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:44.731414    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:11:54.630341 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:54.630301    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:11:54.731663 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:11:54.731628    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:12:04.630363 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:04.630320    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:12:04.731925 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:04.731879    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:12:13.188481 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:13.188442    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 17 22:12:14.731756 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:14.731713    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:12:23.187279 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:23.187253    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:12:24.732243 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:24.732202    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:12:27.029024 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:27.028989    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"]
Sep 17 22:12:27.029552 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:27.029307    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container" containerID="cri-o://c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880" gracePeriod=30
Sep 17 22:12:27.029552 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:27.029353    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kube-rbac-proxy" containerID="cri-o://4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549" gracePeriod=30
Sep 17 22:12:27.046677 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:27.046644    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:27.144373 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:27.144340    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:28.050387 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:28.050355    2583 generic.go:356] "Generic (PLEG): container finished" podID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerID="4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549" exitCode=2
Sep 17 22:12:28.050768 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:28.050422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" event={"ID":"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d","Type":"ContainerDied","Data":"4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549"}
Sep 17 22:12:29.625518 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:29.625479    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 17 22:12:31.976619 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:31.976592    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:12:32.019917 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.019887    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kserve-provision-location\") pod \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") "
Sep 17 22:12:32.020116 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.019941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpzg7\" (UniqueName: \"kubernetes.io/projected/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-api-access-gpzg7\") pod \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") "
Sep 17 22:12:32.020116 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.019985    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-proxy-tls\") pod \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") "
Sep 17 22:12:32.020116 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.020021    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-cabundle-cert\") pod \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") "
Sep 17 22:12:32.020116 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.020096    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-rbac-proxy-sar-config\") pod \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\" (UID: \"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d\") "
Sep 17 22:12:32.020349 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.020240    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" (UID: "dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:12:32.020476 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.020446    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" (UID: "dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:12:32.020531 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.020472    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kserve-provision-location\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:12:32.020568 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.020525    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" (UID: "dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:12:32.022170 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.022122    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" (UID: "dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:12:32.022170 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.022124    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-api-access-gpzg7" (OuterVolumeSpecName: "kube-api-access-gpzg7") pod "dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" (UID: "dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d"). InnerVolumeSpecName "kube-api-access-gpzg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:12:32.076810 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.076728    2583 generic.go:356] "Generic (PLEG): container finished" podID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerID="c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880" exitCode=0
Sep 17 22:12:32.077032 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.076794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" event={"ID":"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d","Type":"ContainerDied","Data":"c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880"}
Sep 17 22:12:32.077032 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.076856    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"
Sep 17 22:12:32.077032 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.076925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h" event={"ID":"dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d","Type":"ContainerDied","Data":"b1319a6c5cb17a60090aca52e687bc675152155eb3a7c30d6d197bcfa62c4128"}
Sep 17 22:12:32.077032 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.076963    2583 scope.go:117] "RemoveContainer" containerID="4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549"
Sep 17 22:12:32.086206 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.086184    2583 scope.go:117] "RemoveContainer" containerID="c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880"
Sep 17 22:12:32.093658 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.093640    2583 scope.go:117] "RemoveContainer" containerID="dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4"
Sep 17 22:12:32.098716 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.098694    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"]
Sep 17 22:12:32.101543 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.101521    2583 scope.go:117] "RemoveContainer" containerID="4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549"
Sep 17 22:12:32.101839 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:12:32.101818    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549\": container with ID starting with 4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549 not found: ID does not exist" containerID="4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549"
Sep 17 22:12:32.101908 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.101853    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549"} err="failed to get container status \"4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549\": rpc error: code = NotFound desc = could not find container \"4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549\": container with ID starting with 4dc0a791b84f88042b0e2620f8189df96635a68433ac488349569de1d3ac0549 not found: ID does not exist"
Sep 17 22:12:32.101908 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.101879    2583 scope.go:117] "RemoveContainer" containerID="c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880"
Sep 17 22:12:32.102194 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:12:32.102169    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880\": container with ID starting with c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880 not found: ID does not exist" containerID="c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880"
Sep 17 22:12:32.102392 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.102200    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880"} err="failed to get container status \"c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880\": rpc error: code = NotFound desc = could not find container \"c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880\": container with ID starting with c9aff6ccf0de39cedb325557598a5cd6beb12997eb7716329b24708835ec6880 not found: ID does not exist"
Sep 17 22:12:32.102392 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.102222    2583 scope.go:117] "RemoveContainer" containerID="dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4"
Sep 17 22:12:32.102866 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:12:32.102815    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4\": container with ID starting with dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4 not found: ID does not exist" containerID="dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4"
Sep 17 22:12:32.102866 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.102848    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4"} err="failed to get container status \"dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4\": rpc error: code = NotFound desc = could not find container \"dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4\": container with ID starting with dbaafbc3e7d31292a759119356dc2d08446eebaeaced4a6b9fb1f0c07c2561b4 not found: ID does not exist"
Sep 17 22:12:32.104641 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.104623    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6cbcf9f68f-fhw5h"]
Sep 17 22:12:32.121302 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.121277    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpzg7\" (UniqueName: \"kubernetes.io/projected/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-api-access-gpzg7\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:12:32.121397 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.121323    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-proxy-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:12:32.121397 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.121334    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-cabundle-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:12:32.121397 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:32.121343    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:12:33.187635 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:33.187603    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" path="/var/lib/kubelet/pods/dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d/volumes"
Sep 17 22:12:34.732826 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:34.732796    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:12:37.276961 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:37.276931    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:37.465918 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:37.465883    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:52.173105 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173072    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"]
Sep 17 22:12:52.173496 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173448    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="storage-initializer"
Sep 17 22:12:52.173496 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173460    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="storage-initializer"
Sep 17 22:12:52.173496 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173481    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container"
Sep 17 22:12:52.173496 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173486    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container"
Sep 17 22:12:52.173496 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173493    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kube-rbac-proxy"
Sep 17 22:12:52.173496 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173499    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kube-rbac-proxy"
Sep 17 22:12:52.173700 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173513    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" containerName="model-chainer-raw-49e3c"
Sep 17 22:12:52.173700 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173520    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" containerName="model-chainer-raw-49e3c"
Sep 17 22:12:52.173700 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173573    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="af5e24a1-3dac-4ffb-9b8f-ddc0bde91c1f" containerName="model-chainer-raw-49e3c"
Sep 17 22:12:52.173700 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173581    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kube-rbac-proxy"
Sep 17 22:12:52.173700 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.173590    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfbd29f0-b8b4-4b8b-90a0-f0d524291c5d" containerName="kserve-container"
Sep 17 22:12:52.176733 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.176713    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:12:52.179211 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.179189    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-5ac84-serving-cert\""
Sep 17 22:12:52.179310 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.179197    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-5ac84-kube-rbac-proxy-sar-config\""
Sep 17 22:12:52.186539 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.186517    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"]
Sep 17 22:12:52.191647 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.191623    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:52.291996 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.291961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/847d4b9c-a197-4832-b7ef-8f98849e6d7c-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-5ac84-769799c559-pm4rh\" (UID: \"847d4b9c-a197-4832-b7ef-8f98849e6d7c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:12:52.292181 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.292021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847d4b9c-a197-4832-b7ef-8f98849e6d7c-proxy-tls\") pod \"model-chainer-raw-hpa-5ac84-769799c559-pm4rh\" (UID: \"847d4b9c-a197-4832-b7ef-8f98849e6d7c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:12:52.314217 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.314185    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:12:52.392935 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.392903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847d4b9c-a197-4832-b7ef-8f98849e6d7c-proxy-tls\") pod \"model-chainer-raw-hpa-5ac84-769799c559-pm4rh\" (UID: \"847d4b9c-a197-4832-b7ef-8f98849e6d7c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:12:52.393094 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.393002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/847d4b9c-a197-4832-b7ef-8f98849e6d7c-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-5ac84-769799c559-pm4rh\" (UID: \"847d4b9c-a197-4832-b7ef-8f98849e6d7c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:12:52.393691 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.393673    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/847d4b9c-a197-4832-b7ef-8f98849e6d7c-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-5ac84-769799c559-pm4rh\" (UID: \"847d4b9c-a197-4832-b7ef-8f98849e6d7c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:12:52.395253 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.395236    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847d4b9c-a197-4832-b7ef-8f98849e6d7c-proxy-tls\") pod \"model-chainer-raw-hpa-5ac84-769799c559-pm4rh\" (UID: \"847d4b9c-a197-4832-b7ef-8f98849e6d7c\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:12:52.487746 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.487661    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:12:52.757368 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:52.757295    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"]
Sep 17 22:12:52.760517 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:12:52.760488    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod847d4b9c_a197_4832_b7ef_8f98849e6d7c.slice/crio-d23c9d32fb202bbec64877726b2c2fa0e48ee5dc1b5fde2bef63fa57ce12ddb3 WatchSource:0}: Error finding container d23c9d32fb202bbec64877726b2c2fa0e48ee5dc1b5fde2bef63fa57ce12ddb3: Status 404 returned error can't find the container with id d23c9d32fb202bbec64877726b2c2fa0e48ee5dc1b5fde2bef63fa57ce12ddb3
Sep 17 22:12:53.176562 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:53.176527    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" event={"ID":"847d4b9c-a197-4832-b7ef-8f98849e6d7c","Type":"ContainerStarted","Data":"7f461b721734bcf12c70471675fb849fdbd1e0f9efc5f3505ee30002fbc45a80"}
Sep 17 22:12:53.177005 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:53.176581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" event={"ID":"847d4b9c-a197-4832-b7ef-8f98849e6d7c","Type":"ContainerStarted","Data":"d23c9d32fb202bbec64877726b2c2fa0e48ee5dc1b5fde2bef63fa57ce12ddb3"}
Sep 17 22:12:53.177005 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:53.176603    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:12:53.193780 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:53.193732    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" podStartSLOduration=1.193716571 podStartE2EDuration="1.193716571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:12:53.192313595 +0000 UTC m=+872.275901543" watchObservedRunningTime="2026-09-17 22:12:53.193716571 +0000 UTC m=+872.277304520"
Sep 17 22:12:59.187223 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:12:59.187193    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:13:02.228276 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:02.228240    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"]
Sep 17 22:13:02.228843 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:02.228452    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" podUID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" containerName="model-chainer-raw-hpa-5ac84" containerID="cri-o://7f461b721734bcf12c70471675fb849fdbd1e0f9efc5f3505ee30002fbc45a80" gracePeriod=30
Sep 17 22:13:02.250236 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:02.250210    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:02.333016 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:02.332986    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"]
Sep 17 22:13:02.333305 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:02.333272    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container" containerID="cri-o://629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0" gracePeriod=30
Sep 17 22:13:02.333527 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:02.333347    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kube-rbac-proxy" containerID="cri-o://ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074" gracePeriod=30
Sep 17 22:13:02.360182 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:02.360129    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:02.433694 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:02.433659    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:02.506808 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:02.506725    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:02.604155 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:02.604117    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:03.226623 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:03.226586    2583 generic.go:356] "Generic (PLEG): container finished" podID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerID="ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074" exitCode=2
Sep 17 22:13:03.226879 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:03.226636    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" event={"ID":"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f","Type":"ContainerDied","Data":"ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074"}
Sep 17 22:13:04.185747 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:04.185663    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" podUID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" containerName="model-chainer-raw-hpa-5ac84" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:04.726909 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:04.726862    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 17 22:13:04.731800 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:04.731765    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 17 22:13:07.078596 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.078571    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:13:07.120760 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.120723    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-rbac-proxy-sar-config\") pod \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") "
Sep 17 22:13:07.120760 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.120769    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kserve-provision-location\") pod \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") "
Sep 17 22:13:07.121022 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.120874    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-proxy-tls\") pod \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") "
Sep 17 22:13:07.121022 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.120939    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-cabundle-cert\") pod \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") "
Sep 17 22:13:07.121022 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.120979    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtdx8\" (UniqueName: \"kubernetes.io/projected/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-api-access-rtdx8\") pod \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\" (UID: \"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f\") "
Sep 17 22:13:07.121216 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.121093    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" (UID: "c32d9318-556e-4c6d-b82f-bf5a83fd0d0f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:13:07.121216 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.121176    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" (UID: "c32d9318-556e-4c6d-b82f-bf5a83fd0d0f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:13:07.121334 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.121283    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" (UID: "c32d9318-556e-4c6d-b82f-bf5a83fd0d0f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:13:07.121334 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.121292    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:13:07.121334 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.121311    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kserve-provision-location\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:13:07.123016 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.122992    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" (UID: "c32d9318-556e-4c6d-b82f-bf5a83fd0d0f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:13:07.123147 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.123095    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-api-access-rtdx8" (OuterVolumeSpecName: "kube-api-access-rtdx8") pod "c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" (UID: "c32d9318-556e-4c6d-b82f-bf5a83fd0d0f"). InnerVolumeSpecName "kube-api-access-rtdx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:13:07.222121 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.222088    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-proxy-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:13:07.222291 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.222184    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-cabundle-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:13:07.222291 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.222214    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtdx8\" (UniqueName: \"kubernetes.io/projected/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f-kube-api-access-rtdx8\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:13:07.248233 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.248199    2583 generic.go:356] "Generic (PLEG): container finished" podID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerID="629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0" exitCode=0
Sep 17 22:13:07.248477 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.248259    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"
Sep 17 22:13:07.248477 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.248276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" event={"ID":"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f","Type":"ContainerDied","Data":"629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0"}
Sep 17 22:13:07.248477 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.248339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc" event={"ID":"c32d9318-556e-4c6d-b82f-bf5a83fd0d0f","Type":"ContainerDied","Data":"1b684b5567e699a9339176869494b4394a8add356c9f67cfe9b8d26d34fc0785"}
Sep 17 22:13:07.248477 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.248367    2583 scope.go:117] "RemoveContainer" containerID="ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074"
Sep 17 22:13:07.258586 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.257334    2583 scope.go:117] "RemoveContainer" containerID="629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0"
Sep 17 22:13:07.266753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.266727    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"]
Sep 17 22:13:07.269642 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.269618    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-5ac84-predictor-758ccdfcb9-sd2kc"]
Sep 17 22:13:07.270701 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.270684    2583 scope.go:117] "RemoveContainer" containerID="4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c"
Sep 17 22:13:07.278391 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.278373    2583 scope.go:117] "RemoveContainer" containerID="ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074"
Sep 17 22:13:07.278707 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:13:07.278684    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074\": container with ID starting with ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074 not found: ID does not exist" containerID="ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074"
Sep 17 22:13:07.278786 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.278715    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074"} err="failed to get container status \"ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074\": rpc error: code = NotFound desc = could not find container \"ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074\": container with ID starting with ed2b027ad7c28e90d21cf886a0df87f53c5155576dc7295f2bb28fa96d216074 not found: ID does not exist"
Sep 17 22:13:07.278786 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.278735    2583 scope.go:117] "RemoveContainer" containerID="629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0"
Sep 17 22:13:07.279002 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:13:07.278985    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0\": container with ID starting with 629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0 not found: ID does not exist" containerID="629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0"
Sep 17 22:13:07.279044 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.279005    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0"} err="failed to get container status \"629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0\": rpc error: code = NotFound desc = could not find container \"629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0\": container with ID starting with 629511221cd0d1c96306e0789673143cdabaf23dfa3e7f48249075ffee1207d0 not found: ID does not exist"
Sep 17 22:13:07.279044 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.279019    2583 scope.go:117] "RemoveContainer" containerID="4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c"
Sep 17 22:13:07.279269 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:13:07.279251    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c\": container with ID starting with 4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c not found: ID does not exist" containerID="4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c"
Sep 17 22:13:07.279330 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:07.279277    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c"} err="failed to get container status \"4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c\": rpc error: code = NotFound desc = could not find container \"4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c\": container with ID starting with 4ee9a936d0f932831bda43078614b7509930dec832307c812f91dbe4f4acf68c not found: ID does not exist"
Sep 17 22:13:09.185002 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:09.184964    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" podUID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" containerName="model-chainer-raw-hpa-5ac84" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:09.186531 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:09.186511    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" path="/var/lib/kubelet/pods/c32d9318-556e-4c6d-b82f-bf5a83fd0d0f/volumes"
Sep 17 22:13:14.185413 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:14.185366    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" podUID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" containerName="model-chainer-raw-hpa-5ac84" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:14.185808 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:14.185494    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:13:19.185012 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:19.184973    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" podUID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" containerName="model-chainer-raw-hpa-5ac84" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:22.442713 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.442673    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:22.531848 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.531808    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"]
Sep 17 22:13:22.532208 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.532194    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container"
Sep 17 22:13:22.532208 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.532209    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container"
Sep 17 22:13:22.532332 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.532223    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="storage-initializer"
Sep 17 22:13:22.532332 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.532229    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="storage-initializer"
Sep 17 22:13:22.532332 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.532247    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kube-rbac-proxy"
Sep 17 22:13:22.532332 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.532254    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kube-rbac-proxy"
Sep 17 22:13:22.532332 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.532305    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kube-rbac-proxy"
Sep 17 22:13:22.532332 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.532313    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c32d9318-556e-4c6d-b82f-bf5a83fd0d0f" containerName="kserve-container"
Sep 17 22:13:22.535884 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.535864    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.540759 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.540736    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-a3db6-kube-rbac-proxy-sar-config\""
Sep 17 22:13:22.540872 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.540851    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-a3db6-predictor-serving-cert\""
Sep 17 22:13:22.540956 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.540940    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:13:22.541522 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.541504    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:13:22.541671 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.541648    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:13:22.563429 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.563396    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"]
Sep 17 22:13:22.651892 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.651862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-cabundle-cert\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.651892 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.651901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.652103 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.651926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d050fef3-5a15-4ba7-80d1-769397e25b9d-kserve-provision-location\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.652103 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.652014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d050fef3-5a15-4ba7-80d1-769397e25b9d-proxy-tls\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.652103 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.652095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlc7p\" (UniqueName: \"kubernetes.io/projected/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-api-access-qlc7p\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.753179 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.753061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.753179 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.753103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d050fef3-5a15-4ba7-80d1-769397e25b9d-kserve-provision-location\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.753386 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.753220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d050fef3-5a15-4ba7-80d1-769397e25b9d-proxy-tls\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.753386 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.753303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlc7p\" (UniqueName: \"kubernetes.io/projected/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-api-access-qlc7p\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.753475 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.753432    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-cabundle-cert\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.753525 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.753507    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d050fef3-5a15-4ba7-80d1-769397e25b9d-kserve-provision-location\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.753802 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.753784    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.753963 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.753946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-cabundle-cert\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.755636 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.755614    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d050fef3-5a15-4ba7-80d1-769397e25b9d-proxy-tls\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.761293 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.761274    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlc7p\" (UniqueName: \"kubernetes.io/projected/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-api-access-qlc7p\") pod \"isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") " pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:22.838980 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.838933    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:13:22.846845 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:22.846815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:23.121916 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:23.121884    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"]
Sep 17 22:13:23.124254 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:13:23.124228    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd050fef3_5a15_4ba7_80d1_769397e25b9d.slice/crio-0fbbc9b324d2576d26cef94de0b97e7f00e701d7e602e53b38738cb2d4e0882a WatchSource:0}: Error finding container 0fbbc9b324d2576d26cef94de0b97e7f00e701d7e602e53b38738cb2d4e0882a: Status 404 returned error can't find the container with id 0fbbc9b324d2576d26cef94de0b97e7f00e701d7e602e53b38738cb2d4e0882a
Sep 17 22:13:23.130398 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:23.130375    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:13:23.135198 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:23.135177    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 22:13:23.139989 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:23.139970    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 22:13:23.140570 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:23.140552    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 22:13:23.145809 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:23.145791    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 22:13:23.323814 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:23.323778    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" event={"ID":"d050fef3-5a15-4ba7-80d1-769397e25b9d","Type":"ContainerStarted","Data":"36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9"}
Sep 17 22:13:23.323982 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:23.323853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" event={"ID":"d050fef3-5a15-4ba7-80d1-769397e25b9d","Type":"ContainerStarted","Data":"0fbbc9b324d2576d26cef94de0b97e7f00e701d7e602e53b38738cb2d4e0882a"}
Sep 17 22:13:24.185641 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:24.185605    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" podUID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" containerName="model-chainer-raw-hpa-5ac84" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:27.343977 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:27.343939    2583 generic.go:356] "Generic (PLEG): container finished" podID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerID="36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9" exitCode=0
Sep 17 22:13:27.344384 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:27.344011    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" event={"ID":"d050fef3-5a15-4ba7-80d1-769397e25b9d","Type":"ContainerDied","Data":"36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9"}
Sep 17 22:13:28.351800 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:28.351765    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" event={"ID":"d050fef3-5a15-4ba7-80d1-769397e25b9d","Type":"ContainerStarted","Data":"6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed"}
Sep 17 22:13:28.352240 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:28.351831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" event={"ID":"d050fef3-5a15-4ba7-80d1-769397e25b9d","Type":"ContainerStarted","Data":"92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2"}
Sep 17 22:13:28.352240 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:28.351855    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" event={"ID":"d050fef3-5a15-4ba7-80d1-769397e25b9d","Type":"ContainerStarted","Data":"46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e"}
Sep 17 22:13:28.352240 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:28.352171    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:28.352240 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:28.352233    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:28.352409 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:28.352267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:28.353846 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:28.353821    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:13:28.354595 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:28.354559    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:28.377531 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:28.377468    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podStartSLOduration=2.780940314 podStartE2EDuration="6.377451606s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.596511292s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:13:28.375654097 +0000 UTC m=+907.459242071" watchObservedRunningTime="2026-09-17 22:13:28.377451606 +0000 UTC m=+907.461039613"
Sep 17 22:13:29.185073 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:29.185035    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" podUID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" containerName="model-chainer-raw-hpa-5ac84" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:29.356947 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:29.356906    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:13:29.357457 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:29.357228    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:32.371061 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:32.371032    2583 generic.go:356] "Generic (PLEG): container finished" podID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" containerID="7f461b721734bcf12c70471675fb849fdbd1e0f9efc5f3505ee30002fbc45a80" exitCode=0
Sep 17 22:13:32.371403 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:32.371076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" event={"ID":"847d4b9c-a197-4832-b7ef-8f98849e6d7c","Type":"ContainerDied","Data":"7f461b721734bcf12c70471675fb849fdbd1e0f9efc5f3505ee30002fbc45a80"}
Sep 17 22:13:32.371403 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:32.371113    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh" event={"ID":"847d4b9c-a197-4832-b7ef-8f98849e6d7c","Type":"ContainerDied","Data":"d23c9d32fb202bbec64877726b2c2fa0e48ee5dc1b5fde2bef63fa57ce12ddb3"}
Sep 17 22:13:32.371403 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:32.371130    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d23c9d32fb202bbec64877726b2c2fa0e48ee5dc1b5fde2bef63fa57ce12ddb3"
Sep 17 22:13:32.371403 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:32.371268    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:13:32.443496 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:32.443463    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/847d4b9c-a197-4832-b7ef-8f98849e6d7c-openshift-service-ca-bundle\") pod \"847d4b9c-a197-4832-b7ef-8f98849e6d7c\" (UID: \"847d4b9c-a197-4832-b7ef-8f98849e6d7c\") "
Sep 17 22:13:32.443679 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:32.443505    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847d4b9c-a197-4832-b7ef-8f98849e6d7c-proxy-tls\") pod \"847d4b9c-a197-4832-b7ef-8f98849e6d7c\" (UID: \"847d4b9c-a197-4832-b7ef-8f98849e6d7c\") "
Sep 17 22:13:32.443824 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:32.443797    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847d4b9c-a197-4832-b7ef-8f98849e6d7c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "847d4b9c-a197-4832-b7ef-8f98849e6d7c" (UID: "847d4b9c-a197-4832-b7ef-8f98849e6d7c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:13:32.445669 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:32.445648    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847d4b9c-a197-4832-b7ef-8f98849e6d7c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "847d4b9c-a197-4832-b7ef-8f98849e6d7c" (UID: "847d4b9c-a197-4832-b7ef-8f98849e6d7c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:13:32.544906 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:32.544816    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/847d4b9c-a197-4832-b7ef-8f98849e6d7c-openshift-service-ca-bundle\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:13:32.544906 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:32.544846    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/847d4b9c-a197-4832-b7ef-8f98849e6d7c-proxy-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:13:33.375456 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:33.375417    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"
Sep 17 22:13:33.391698 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:33.391668    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"]
Sep 17 22:13:33.397066 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:33.397043    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-5ac84-769799c559-pm4rh"]
Sep 17 22:13:34.361315 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:34.361287    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:13:34.362001 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:34.361965    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:13:34.362249 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:34.362223    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:35.186786 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:35.186755    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" path="/var/lib/kubelet/pods/847d4b9c-a197-4832-b7ef-8f98849e6d7c/volumes"
Sep 17 22:13:44.362366 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:44.362325    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:13:44.362923 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:44.362897    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:13:54.362587 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:54.362537    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:13:54.363079 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:13:54.363047    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:14:04.362343 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:04.362293    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:14:04.362769 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:04.362710    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:14:14.362184 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:14.362118    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:14:14.362702 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:14.362607    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:14:24.362268 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:24.362224    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:14:24.362807 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:24.362779    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:14:32.472946 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:32.472910    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:34.362074 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:34.361981    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:14:34.362498 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:34.362429    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:14:34.442406 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:34.442370    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:39.180556 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:39.180511    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:14:39.180999 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:39.180889    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:14:49.185537 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:49.185506    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:14:49.186098 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:49.185598    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:14:50.572808 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:50.572769    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:50.644782 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:50.644748    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:55.720956 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:55.720921    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:55.928472 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:55.928439    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:57.637302 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:57.637264    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"]
Sep 17 22:14:57.637749 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:57.637620    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kube-rbac-proxy" containerID="cri-o://92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2" gracePeriod=30
Sep 17 22:14:57.637749 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:57.637598    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" containerID="cri-o://6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed" gracePeriod=30
Sep 17 22:14:57.638499 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:57.638091    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" containerID="cri-o://46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e" gracePeriod=30
Sep 17 22:14:57.663446 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:57.663413    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:57.759586 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:57.759552    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:57.823477 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:57.823435    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:58.429725 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:58.429676    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:14:58.760556 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:58.760473    2583 generic.go:356] "Generic (PLEG): container finished" podID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerID="92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2" exitCode=2
Sep 17 22:14:58.760556 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:58.760510    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" event={"ID":"d050fef3-5a15-4ba7-80d1-769397e25b9d","Type":"ContainerDied","Data":"92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2"}
Sep 17 22:14:59.180614 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:59.180567    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:14:59.180960 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:59.180923    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:14:59.358000 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:14:59.357962    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 22:15:02.782035 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:02.781953    2583 generic.go:356] "Generic (PLEG): container finished" podID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerID="46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e" exitCode=0
Sep 17 22:15:02.782035 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:02.782004    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" event={"ID":"d050fef3-5a15-4ba7-80d1-769397e25b9d","Type":"ContainerDied","Data":"46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e"}
Sep 17 22:15:04.357952 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:04.357913    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 22:15:05.793151 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:05.793066    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:15:09.180594 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:09.180548    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:15:09.181122 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:09.180947    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:15:09.357825 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:09.357781    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 22:15:09.358003 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:09.357914    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:15:10.958494 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:10.958464    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:15:11.029369 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:11.029338    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:15:14.357009 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:14.356958    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 22:15:19.180908 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:19.180862    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 22:15:19.181395 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:19.181331    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:15:19.186466 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:19.186443    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:15:19.186598 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:19.186530    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:15:19.357921 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:19.357880    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 22:15:21.700676 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:21.700638    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:15:24.357983 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:24.357938    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 17 22:15:27.794270 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.794242    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:15:27.866101 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.866068    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d050fef3-5a15-4ba7-80d1-769397e25b9d-proxy-tls\") pod \"d050fef3-5a15-4ba7-80d1-769397e25b9d\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") "
Sep 17 22:15:27.866101 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.866108    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d050fef3-5a15-4ba7-80d1-769397e25b9d-kserve-provision-location\") pod \"d050fef3-5a15-4ba7-80d1-769397e25b9d\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") "
Sep 17 22:15:27.866316 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.866126    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-rbac-proxy-sar-config\") pod \"d050fef3-5a15-4ba7-80d1-769397e25b9d\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") "
Sep 17 22:15:27.866316 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.866175    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlc7p\" (UniqueName: \"kubernetes.io/projected/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-api-access-qlc7p\") pod \"d050fef3-5a15-4ba7-80d1-769397e25b9d\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") "
Sep 17 22:15:27.866316 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.866213    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-cabundle-cert\") pod \"d050fef3-5a15-4ba7-80d1-769397e25b9d\" (UID: \"d050fef3-5a15-4ba7-80d1-769397e25b9d\") "
Sep 17 22:15:27.866550 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.866454    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d050fef3-5a15-4ba7-80d1-769397e25b9d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d050fef3-5a15-4ba7-80d1-769397e25b9d" (UID: "d050fef3-5a15-4ba7-80d1-769397e25b9d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:15:27.866637 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.866530    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d050fef3-5a15-4ba7-80d1-769397e25b9d" (UID: "d050fef3-5a15-4ba7-80d1-769397e25b9d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:15:27.866680 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.866633    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d050fef3-5a15-4ba7-80d1-769397e25b9d" (UID: "d050fef3-5a15-4ba7-80d1-769397e25b9d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:15:27.868300 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.868283    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d050fef3-5a15-4ba7-80d1-769397e25b9d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d050fef3-5a15-4ba7-80d1-769397e25b9d" (UID: "d050fef3-5a15-4ba7-80d1-769397e25b9d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:15:27.868388 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.868364    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-api-access-qlc7p" (OuterVolumeSpecName: "kube-api-access-qlc7p") pod "d050fef3-5a15-4ba7-80d1-769397e25b9d" (UID: "d050fef3-5a15-4ba7-80d1-769397e25b9d"). InnerVolumeSpecName "kube-api-access-qlc7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:15:27.910796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.910715    2583 generic.go:356] "Generic (PLEG): container finished" podID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerID="6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed" exitCode=0
Sep 17 22:15:27.910796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.910783    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"
Sep 17 22:15:27.910958 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.910801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" event={"ID":"d050fef3-5a15-4ba7-80d1-769397e25b9d","Type":"ContainerDied","Data":"6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed"}
Sep 17 22:15:27.910958 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.910893    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9" event={"ID":"d050fef3-5a15-4ba7-80d1-769397e25b9d","Type":"ContainerDied","Data":"0fbbc9b324d2576d26cef94de0b97e7f00e701d7e602e53b38738cb2d4e0882a"}
Sep 17 22:15:27.910958 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.910924    2583 scope.go:117] "RemoveContainer" containerID="6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed"
Sep 17 22:15:27.919527 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.919513    2583 scope.go:117] "RemoveContainer" containerID="92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2"
Sep 17 22:15:27.926954 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.926935    2583 scope.go:117] "RemoveContainer" containerID="46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e"
Sep 17 22:15:27.933225 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.933198    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"]
Sep 17 22:15:27.934725 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.934700    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-a3db6-predictor-65555b4bc6-rgcj9"]
Sep 17 22:15:27.935922 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.935898    2583 scope.go:117] "RemoveContainer" containerID="36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9"
Sep 17 22:15:27.943753 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.943735    2583 scope.go:117] "RemoveContainer" containerID="6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed"
Sep 17 22:15:27.944078 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:15:27.944055    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed\": container with ID starting with 6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed not found: ID does not exist" containerID="6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed"
Sep 17 22:15:27.944191 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.944089    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed"} err="failed to get container status \"6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed\": rpc error: code = NotFound desc = could not find container \"6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed\": container with ID starting with 6e2a657390533592b51b4faa82dc5ed7e55c88a0d53016f263cbacb04ee726ed not found: ID does not exist"
Sep 17 22:15:27.944191 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.944116    2583 scope.go:117] "RemoveContainer" containerID="92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2"
Sep 17 22:15:27.944410 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:15:27.944392    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2\": container with ID starting with 92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2 not found: ID does not exist" containerID="92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2"
Sep 17 22:15:27.944450 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.944418    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2"} err="failed to get container status \"92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2\": rpc error: code = NotFound desc = could not find container \"92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2\": container with ID starting with 92ba2ca590f8b60a8b834e10474ecd9fb245f313d2b57a0d154ffc121b10a2f2 not found: ID does not exist"
Sep 17 22:15:27.944450 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.944434    2583 scope.go:117] "RemoveContainer" containerID="46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e"
Sep 17 22:15:27.944657 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:15:27.944639    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e\": container with ID starting with 46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e not found: ID does not exist" containerID="46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e"
Sep 17 22:15:27.944721 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.944664    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e"} err="failed to get container status \"46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e\": rpc error: code = NotFound desc = could not find container \"46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e\": container with ID starting with 46f1f85a9ac2a0dc9fb2f86d82b23108f68dacc48eb34518f8ee987e7377148e not found: ID does not exist"
Sep 17 22:15:27.944721 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.944685    2583 scope.go:117] "RemoveContainer" containerID="36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9"
Sep 17 22:15:27.944901 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:15:27.944886    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9\": container with ID starting with 36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9 not found: ID does not exist" containerID="36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9"
Sep 17 22:15:27.944940 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.944906    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9"} err="failed to get container status \"36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9\": rpc error: code = NotFound desc = could not find container \"36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9\": container with ID starting with 36f590eac336a67045aa5a2594d0f1ef1aa6f1c57ca098281f37223e46e897b9 not found: ID does not exist"
Sep 17 22:15:27.967192 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.967163    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlc7p\" (UniqueName: \"kubernetes.io/projected/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-api-access-qlc7p\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:15:27.967192 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.967189    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-cabundle-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:15:27.967327 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.967200    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d050fef3-5a15-4ba7-80d1-769397e25b9d-proxy-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:15:27.967327 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.967208    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d050fef3-5a15-4ba7-80d1-769397e25b9d-kserve-provision-location\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:15:27.967327 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:27.967216    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d050fef3-5a15-4ba7-80d1-769397e25b9d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:15:29.186972 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:15:29.186934    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" path="/var/lib/kubelet/pods/d050fef3-5a15-4ba7-80d1-769397e25b9d/volumes"
Sep 17 22:16:25.011657 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:16:25.011616    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:25.071410 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:16:25.071377    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:41.132798 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:16:41.132764    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:46.259292 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:16:46.259250    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:16:46.693490 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:16:46.693457    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:05.301323 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:05.301280    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:24.721737 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:24.721703    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:47.893282 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:47.893248    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:53.055260 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.055229    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:53.110961 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.110924    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"]
Sep 17 22:17:53.111308 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111294    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent"
Sep 17 22:17:53.111308 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111309    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111321    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" containerName="model-chainer-raw-hpa-5ac84"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111327    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" containerName="model-chainer-raw-hpa-5ac84"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111334    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="storage-initializer"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111340    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="storage-initializer"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111352    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111357    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111367    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kube-rbac-proxy"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111372    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kube-rbac-proxy"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111423    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kserve-container"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111432    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="kube-rbac-proxy"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111437    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="847d4b9c-a197-4832-b7ef-8f98849e6d7c" containerName="model-chainer-raw-hpa-5ac84"
Sep 17 22:17:53.111442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.111445    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d050fef3-5a15-4ba7-80d1-769397e25b9d" containerName="agent"
Sep 17 22:17:53.114974 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.114948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.117761 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.117736    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 22:17:53.117761 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.117753    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:17:53.118000 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.117974    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:17:53.118120 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.118009    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:17:53.118291 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.118270    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:17:53.118414 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.118326    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 22:17:53.118769 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.118749    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:17:53.123572 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.123549    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"]
Sep 17 22:17:53.143443 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.143397    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:53.196120 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.196089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.196120 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.196122    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.196337 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.196245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzt25\" (UniqueName: \"kubernetes.io/projected/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-api-access-vzt25\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.196337 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.196298    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.196420 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.196353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.234428 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.234400    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:53.297362 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.297323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzt25\" (UniqueName: \"kubernetes.io/projected/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-api-access-vzt25\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.297520 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.297380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.297520 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.297417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.297520 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.297484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.297520 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.297506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.297989 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.297962    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.298084 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.298029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.298232 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.298213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.300019 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.299996    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.305987 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.305930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzt25\" (UniqueName: \"kubernetes.io/projected/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-api-access-vzt25\") pod \"isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.322155 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.322121    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:17:53.427931 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.427897    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:53.704115 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:53.704060    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"]
Sep 17 22:17:53.706177 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:17:53.706152    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce0e5f6_8a94_4365_ba06_f1e7e0cbe85d.slice/crio-4e0744c068c6157a71e734a05a1eb0576f84a749e409aec1254a3c1036002349 WatchSource:0}: Error finding container 4e0744c068c6157a71e734a05a1eb0576f84a749e409aec1254a3c1036002349: Status 404 returned error can't find the container with id 4e0744c068c6157a71e734a05a1eb0576f84a749e409aec1254a3c1036002349
Sep 17 22:17:54.587043 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:54.587000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" event={"ID":"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d","Type":"ContainerStarted","Data":"c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507"}
Sep 17 22:17:54.587521 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:54.587057    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" event={"ID":"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d","Type":"ContainerStarted","Data":"4e0744c068c6157a71e734a05a1eb0576f84a749e409aec1254a3c1036002349"}
Sep 17 22:17:57.601256 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:57.601219    2583 generic.go:356] "Generic (PLEG): container finished" podID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerID="c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507" exitCode=0
Sep 17 22:17:57.601663 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:57.601289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" event={"ID":"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d","Type":"ContainerDied","Data":"c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507"}
Sep 17 22:17:58.610219 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:58.610184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" event={"ID":"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d","Type":"ContainerStarted","Data":"73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08"}
Sep 17 22:17:58.610645 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:58.610249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" event={"ID":"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d","Type":"ContainerStarted","Data":"0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7"}
Sep 17 22:17:58.610645 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:58.610546    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:58.610724 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:58.610709    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:17:58.612015 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:58.611992    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:17:58.630089 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:58.630038    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podStartSLOduration=1.94454056 podStartE2EDuration="5.63002552s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.68548496s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:17:58.627791829 +0000 UTC m=+1177.711379796" watchObservedRunningTime="2026-09-17 22:17:58.63002552 +0000 UTC m=+1177.713613468"
Sep 17 22:17:59.615064 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:17:59.615020    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:18:04.619651 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:04.619619    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:18:04.620163 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:04.620111    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:18:14.620644 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:14.620599    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:18:16.368558 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:16.368512    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:18:23.163487 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:23.163458    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 22:18:23.168758 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:23.168738    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 22:18:23.170200 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:23.170182    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 22:18:23.175084 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:23.175068    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 22:18:24.620397 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:24.620359    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:18:34.620796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:34.620756    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:18:44.620255 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:44.620211    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:18:54.620605 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:18:54.620567    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:19:04.621101 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:04.621006    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:19:14.621416 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:14.621385    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:19:23.148642 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:23.148604    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"]
Sep 17 22:19:23.149059 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:23.148907    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container" containerID="cri-o://0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7" gracePeriod=30
Sep 17 22:19:23.149059 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:23.148933    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kube-rbac-proxy" containerID="cri-o://73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08" gracePeriod=30
Sep 17 22:19:23.163094 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:23.163068    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:19:23.206825 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:23.206785    2583 scope.go:117] "RemoveContainer" containerID="7f461b721734bcf12c70471675fb849fdbd1e0f9efc5f3505ee30002fbc45a80"
Sep 17 22:19:23.228251 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:23.228219    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:19:23.994253 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:23.994216    2583 generic.go:356] "Generic (PLEG): container finished" podID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerID="73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08" exitCode=2
Sep 17 22:19:23.994444 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:23.994293    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" event={"ID":"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d","Type":"ContainerDied","Data":"73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08"}
Sep 17 22:19:24.615566 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:24.615521    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 17 22:19:24.620507 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:24.620474    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 17 22:19:27.695176 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.695149    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:19:27.790483 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.790407    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kserve-provision-location\") pod \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") "
Sep 17 22:19:27.790483 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.790443    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-proxy-tls\") pod \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") "
Sep 17 22:19:27.790483 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.790469    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzt25\" (UniqueName: \"kubernetes.io/projected/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-api-access-vzt25\") pod \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") "
Sep 17 22:19:27.790731 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.790529    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-cabundle-cert\") pod \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") "
Sep 17 22:19:27.790731 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.790578    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-rbac-proxy-sar-config\") pod \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\" (UID: \"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d\") "
Sep 17 22:19:27.790731 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.790711    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" (UID: "8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:19:27.790900 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.790827    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kserve-provision-location\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:19:27.790959 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.790931    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" (UID: "8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:27.791011 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.790971    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" (UID: "8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:19:27.792562 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.792542    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-api-access-vzt25" (OuterVolumeSpecName: "kube-api-access-vzt25") pod "8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" (UID: "8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d"). InnerVolumeSpecName "kube-api-access-vzt25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:19:27.792632 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.792561    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" (UID: "8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:19:27.892028 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.891991    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-proxy-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:19:27.892028 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.892022    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzt25\" (UniqueName: \"kubernetes.io/projected/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-api-access-vzt25\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:19:27.892028 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.892035    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-cabundle-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:19:27.892286 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:27.892048    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:19:28.015912 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.015879    2583 generic.go:356] "Generic (PLEG): container finished" podID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerID="0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7" exitCode=0
Sep 17 22:19:28.016101 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.015939    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"
Sep 17 22:19:28.016101 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.015948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" event={"ID":"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d","Type":"ContainerDied","Data":"0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7"}
Sep 17 22:19:28.016101 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.016010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz" event={"ID":"8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d","Type":"ContainerDied","Data":"4e0744c068c6157a71e734a05a1eb0576f84a749e409aec1254a3c1036002349"}
Sep 17 22:19:28.016101 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.016046    2583 scope.go:117] "RemoveContainer" containerID="73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08"
Sep 17 22:19:28.029196 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.029177    2583 scope.go:117] "RemoveContainer" containerID="0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7"
Sep 17 22:19:28.036732 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.036714    2583 scope.go:117] "RemoveContainer" containerID="c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507"
Sep 17 22:19:28.041181 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.041094    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"]
Sep 17 22:19:28.044751 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.044728    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79d6f5c6d9-sj6gz"]
Sep 17 22:19:28.044813 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.044762    2583 scope.go:117] "RemoveContainer" containerID="73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08"
Sep 17 22:19:28.045050 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:19:28.045030    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08\": container with ID starting with 73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08 not found: ID does not exist" containerID="73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08"
Sep 17 22:19:28.045112 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.045058    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08"} err="failed to get container status \"73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08\": rpc error: code = NotFound desc = could not find container \"73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08\": container with ID starting with 73631ce862f4fece8a36ea4c0f7269fb7f2de8004580c28f0297b741af6dda08 not found: ID does not exist"
Sep 17 22:19:28.045112 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.045076    2583 scope.go:117] "RemoveContainer" containerID="0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7"
Sep 17 22:19:28.045388 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:19:28.045371    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7\": container with ID starting with 0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7 not found: ID does not exist" containerID="0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7"
Sep 17 22:19:28.045437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.045394    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7"} err="failed to get container status \"0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7\": rpc error: code = NotFound desc = could not find container \"0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7\": container with ID starting with 0be92ace48fd16c9e348f79864f9789c65c6559e4e3b19656c70be9013698bf7 not found: ID does not exist"
Sep 17 22:19:28.045437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.045414    2583 scope.go:117] "RemoveContainer" containerID="c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507"
Sep 17 22:19:28.045676 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:19:28.045656    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507\": container with ID starting with c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507 not found: ID does not exist" containerID="c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507"
Sep 17 22:19:28.045745 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.045686    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507"} err="failed to get container status \"c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507\": rpc error: code = NotFound desc = could not find container \"c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507\": container with ID starting with c29b29186aee3522507782d4dfd51a22bd9e9c5913a070234d473c278c8fa507 not found: ID does not exist"
Sep 17 22:19:28.292467 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.292372    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:19:28.369932 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.369905    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:19:28.416987 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.416959    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"]
Sep 17 22:19:28.417338 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.417325    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="storage-initializer"
Sep 17 22:19:28.417393 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.417340    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="storage-initializer"
Sep 17 22:19:28.417393 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.417354    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container"
Sep 17 22:19:28.417393 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.417360    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container"
Sep 17 22:19:28.417393 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.417367    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kube-rbac-proxy"
Sep 17 22:19:28.417393 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.417372    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kube-rbac-proxy"
Sep 17 22:19:28.417548 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.417425    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kserve-container"
Sep 17 22:19:28.417548 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.417437    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" containerName="kube-rbac-proxy"
Sep 17 22:19:28.421903 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.421883    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.424337 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.424310    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:19:28.424337 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.424318    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:19:28.424505 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.424453    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:19:28.424672 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.424647    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 22:19:28.424790 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.424721    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:19:28.424790 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.424720    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 22:19:28.424790 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.424772    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:19:28.431980 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.431956    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"]
Sep 17 22:19:28.496963 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.496935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76xs2\" (UniqueName: \"kubernetes.io/projected/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-api-access-76xs2\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.497100 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.497049    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85ccf83b-5c9a-4619-81e8-9edf7604330b-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.497100 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.497078    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.497200 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.497103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.497200 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.497183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ccf83b-5c9a-4619-81e8-9edf7604330b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.552222 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.552105    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:19:28.598120 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.598086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ccf83b-5c9a-4619-81e8-9edf7604330b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.598120 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.598124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76xs2\" (UniqueName: \"kubernetes.io/projected/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-api-access-76xs2\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.598324 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.598223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85ccf83b-5c9a-4619-81e8-9edf7604330b-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.598324 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.598248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.598324 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.598270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.598535 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.598514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ccf83b-5c9a-4619-81e8-9edf7604330b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.598850 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.598825    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-cabundle-cert\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.598962 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.598913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.600989 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.600971    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85ccf83b-5c9a-4619-81e8-9edf7604330b-proxy-tls\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.605505 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.605483    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76xs2\" (UniqueName: \"kubernetes.io/projected/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-api-access-76xs2\") pod \"isvc-canary-promote-predictor-567686cc76-j5q95\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:28.652296 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.652258    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:19:28.734131 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:28.734104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:29.005867 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:29.005841    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"]
Sep 17 22:19:29.007548 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:19:29.007522    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ccf83b_5c9a_4619_81e8_9edf7604330b.slice/crio-13a5b71e121fdabe3fef4408c351e9a9cf33761c1bccecec6721871f67ccaa3b WatchSource:0}: Error finding container 13a5b71e121fdabe3fef4408c351e9a9cf33761c1bccecec6721871f67ccaa3b: Status 404 returned error can't find the container with id 13a5b71e121fdabe3fef4408c351e9a9cf33761c1bccecec6721871f67ccaa3b
Sep 17 22:19:29.021732 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:29.021706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" event={"ID":"85ccf83b-5c9a-4619-81e8-9edf7604330b","Type":"ContainerStarted","Data":"13a5b71e121fdabe3fef4408c351e9a9cf33761c1bccecec6721871f67ccaa3b"}
Sep 17 22:19:29.186108 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:29.186068    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d" path="/var/lib/kubelet/pods/8ce0e5f6-8a94-4365-ba06-f1e7e0cbe85d/volumes"
Sep 17 22:19:30.028512 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:30.028465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" event={"ID":"85ccf83b-5c9a-4619-81e8-9edf7604330b","Type":"ContainerStarted","Data":"1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c"}
Sep 17 22:19:33.043356 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:33.043268    2583 generic.go:356] "Generic (PLEG): container finished" podID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerID="1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c" exitCode=0
Sep 17 22:19:33.043356 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:33.043343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" event={"ID":"85ccf83b-5c9a-4619-81e8-9edf7604330b","Type":"ContainerDied","Data":"1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c"}
Sep 17 22:19:34.050084 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:34.050051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" event={"ID":"85ccf83b-5c9a-4619-81e8-9edf7604330b","Type":"ContainerStarted","Data":"13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4"}
Sep 17 22:19:34.050482 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:34.050162    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" event={"ID":"85ccf83b-5c9a-4619-81e8-9edf7604330b","Type":"ContainerStarted","Data":"e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39"}
Sep 17 22:19:34.050482 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:34.050366    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:34.070034 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:34.069985    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podStartSLOduration=2.401751815 podStartE2EDuration="6.069969291s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.668217476s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:19:34.068661528 +0000 UTC m=+1273.152249498" watchObservedRunningTime="2026-09-17 22:19:34.069969291 +0000 UTC m=+1273.153557239"
Sep 17 22:19:35.054260 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:35.054212    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:35.055434 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:35.055394    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:19:36.059219 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:36.059176    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:19:41.064620 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:41.064589    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:19:41.065094 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:41.065069    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:19:42.165898 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:42.165850    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:19:51.065822 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:19:51.065780    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:20:01.065360 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:01.065320    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:20:11.065845 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:11.065802    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:20:21.065365 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:21.065328    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:20:31.065341 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:31.065299    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:20:41.065382 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:41.065291    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:20:51.065727 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:51.065694    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:20:58.441546 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:58.441508    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"]
Sep 17 22:20:58.442040 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:58.441799    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container" containerID="cri-o://e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39" gracePeriod=30
Sep 17 22:20:58.442040 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:58.441837    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kube-rbac-proxy" containerID="cri-o://13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4" gracePeriod=30
Sep 17 22:20:58.466413 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:58.466388    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:20:59.436442 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:59.436410    2583 generic.go:356] "Generic (PLEG): container finished" podID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerID="13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4" exitCode=2
Sep 17 22:20:59.436616 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:20:59.436470    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" event={"ID":"85ccf83b-5c9a-4619-81e8-9edf7604330b","Type":"ContainerDied","Data":"13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4"}
Sep 17 22:21:01.060346 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:01.060296    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 17 22:21:01.065762 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:01.065735    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 17 22:21:02.982879 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:02.982849    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:21:03.090439 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.090353    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-rbac-proxy-sar-config\") pod \"85ccf83b-5c9a-4619-81e8-9edf7604330b\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") "
Sep 17 22:21:03.090439 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.090405    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ccf83b-5c9a-4619-81e8-9edf7604330b-kserve-provision-location\") pod \"85ccf83b-5c9a-4619-81e8-9edf7604330b\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") "
Sep 17 22:21:03.090439 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.090428    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-cabundle-cert\") pod \"85ccf83b-5c9a-4619-81e8-9edf7604330b\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") "
Sep 17 22:21:03.090664 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.090484    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85ccf83b-5c9a-4619-81e8-9edf7604330b-proxy-tls\") pod \"85ccf83b-5c9a-4619-81e8-9edf7604330b\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") "
Sep 17 22:21:03.090664 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.090533    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76xs2\" (UniqueName: \"kubernetes.io/projected/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-api-access-76xs2\") pod \"85ccf83b-5c9a-4619-81e8-9edf7604330b\" (UID: \"85ccf83b-5c9a-4619-81e8-9edf7604330b\") "
Sep 17 22:21:03.090786 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.090757    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85ccf83b-5c9a-4619-81e8-9edf7604330b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85ccf83b-5c9a-4619-81e8-9edf7604330b" (UID: "85ccf83b-5c9a-4619-81e8-9edf7604330b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:21:03.090828 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.090787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85ccf83b-5c9a-4619-81e8-9edf7604330b" (UID: "85ccf83b-5c9a-4619-81e8-9edf7604330b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:21:03.090867 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.090836    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85ccf83b-5c9a-4619-81e8-9edf7604330b" (UID: "85ccf83b-5c9a-4619-81e8-9edf7604330b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:21:03.092687 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.092656    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-api-access-76xs2" (OuterVolumeSpecName: "kube-api-access-76xs2") pod "85ccf83b-5c9a-4619-81e8-9edf7604330b" (UID: "85ccf83b-5c9a-4619-81e8-9edf7604330b"). InnerVolumeSpecName "kube-api-access-76xs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:21:03.092687 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.092663    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ccf83b-5c9a-4619-81e8-9edf7604330b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85ccf83b-5c9a-4619-81e8-9edf7604330b" (UID: "85ccf83b-5c9a-4619-81e8-9edf7604330b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:21:03.191394 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.191358    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85ccf83b-5c9a-4619-81e8-9edf7604330b-kserve-provision-location\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:21:03.191394 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.191387    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-cabundle-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:21:03.191394 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.191401    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85ccf83b-5c9a-4619-81e8-9edf7604330b-proxy-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:21:03.191632 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.191412    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76xs2\" (UniqueName: \"kubernetes.io/projected/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-api-access-76xs2\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:21:03.191632 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.191424    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85ccf83b-5c9a-4619-81e8-9edf7604330b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:21:03.456065 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.456031    2583 generic.go:356] "Generic (PLEG): container finished" podID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerID="e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39" exitCode=0
Sep 17 22:21:03.456249 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.456082    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" event={"ID":"85ccf83b-5c9a-4619-81e8-9edf7604330b","Type":"ContainerDied","Data":"e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39"}
Sep 17 22:21:03.456249 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.456087    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"
Sep 17 22:21:03.456249 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.456130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95" event={"ID":"85ccf83b-5c9a-4619-81e8-9edf7604330b","Type":"ContainerDied","Data":"13a5b71e121fdabe3fef4408c351e9a9cf33761c1bccecec6721871f67ccaa3b"}
Sep 17 22:21:03.456249 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.456186    2583 scope.go:117] "RemoveContainer" containerID="13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4"
Sep 17 22:21:03.464420 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.464402    2583 scope.go:117] "RemoveContainer" containerID="e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39"
Sep 17 22:21:03.471468 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.471453    2583 scope.go:117] "RemoveContainer" containerID="1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c"
Sep 17 22:21:03.476336 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.476316    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"]
Sep 17 22:21:03.480164 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.480125    2583 scope.go:117] "RemoveContainer" containerID="13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4"
Sep 17 22:21:03.481942 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.481923    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-567686cc76-j5q95"]
Sep 17 22:21:03.482048 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:21:03.481935    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4\": container with ID starting with 13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4 not found: ID does not exist" containerID="13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4"
Sep 17 22:21:03.482048 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.481969    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4"} err="failed to get container status \"13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4\": rpc error: code = NotFound desc = could not find container \"13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4\": container with ID starting with 13a21516ffd6a860e19c17facdae60a09e10c38268511b921c2a40271f4f39d4 not found: ID does not exist"
Sep 17 22:21:03.482048 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.481993    2583 scope.go:117] "RemoveContainer" containerID="e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39"
Sep 17 22:21:03.482364 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:21:03.482339    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39\": container with ID starting with e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39 not found: ID does not exist" containerID="e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39"
Sep 17 22:21:03.482416 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.482363    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39"} err="failed to get container status \"e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39\": rpc error: code = NotFound desc = could not find container \"e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39\": container with ID starting with e5dff4e9bcec32379bd6261c92c8fdfc1ddd3e6e393ed5ae0c726370d67cca39 not found: ID does not exist"
Sep 17 22:21:03.482416 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.482378    2583 scope.go:117] "RemoveContainer" containerID="1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c"
Sep 17 22:21:03.482603 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:21:03.482581    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c\": container with ID starting with 1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c not found: ID does not exist" containerID="1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c"
Sep 17 22:21:03.482661 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:03.482607    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c"} err="failed to get container status \"1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c\": rpc error: code = NotFound desc = could not find container \"1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c\": container with ID starting with 1effdf9362e49752c627afa2194fce2b0fdf06778bfb24c385ac5b399b10ef6c not found: ID does not exist"
Sep 17 22:21:05.185791 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:21:05.185757    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" path="/var/lib/kubelet/pods/85ccf83b-5c9a-4619-81e8-9edf7604330b/volumes"
Sep 17 22:22:28.722207 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:28.722167    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:42.740058 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:42.740021    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:43.306016 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:43.305966    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:52.749760 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.749724    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:52.788812 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.788780    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"]
Sep 17 22:22:52.789127 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.789114    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container"
Sep 17 22:22:52.789193 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.789129    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container"
Sep 17 22:22:52.789193 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.789154    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kube-rbac-proxy"
Sep 17 22:22:52.789193 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.789160    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kube-rbac-proxy"
Sep 17 22:22:52.789193 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.789178    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="storage-initializer"
Sep 17 22:22:52.789193 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.789183    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="storage-initializer"
Sep 17 22:22:52.789345 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.789237    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kube-rbac-proxy"
Sep 17 22:22:52.789345 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.789245    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ccf83b-5c9a-4619-81e8-9edf7604330b" containerName="kserve-container"
Sep 17 22:22:52.792509 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.792491    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:52.794997 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.794970    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 22:22:52.795115 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.795005    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 22:22:52.795115 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.795092    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 22:22:52.795313 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.795296    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 22:22:52.795387 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.795349    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 17 22:22:52.795387 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.795367    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-nzt4f\""
Sep 17 22:22:52.795497 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.795419    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 22:22:52.805005 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.804981    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"]
Sep 17 22:22:52.931018 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.930956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:52.931234 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.931079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3138b353-4c31-4585-82c5-d476bd07f04e-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:52.931286 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.931233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3138b353-4c31-4585-82c5-d476bd07f04e-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:52.931328 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.931286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:52.931373 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:52.931345    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbwq2\" (UniqueName: \"kubernetes.io/projected/3138b353-4c31-4585-82c5-d476bd07f04e-kube-api-access-hbwq2\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.032597 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.032515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbwq2\" (UniqueName: \"kubernetes.io/projected/3138b353-4c31-4585-82c5-d476bd07f04e-kube-api-access-hbwq2\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.032597 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.032557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.032796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.032679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3138b353-4c31-4585-82c5-d476bd07f04e-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.032796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.032760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3138b353-4c31-4585-82c5-d476bd07f04e-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.032796 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.032794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.033117 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.033098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.033229 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.033174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3138b353-4c31-4585-82c5-d476bd07f04e-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.033400 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.033382    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.035477 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.035456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3138b353-4c31-4585-82c5-d476bd07f04e-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.040830 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.040807    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbwq2\" (UniqueName: \"kubernetes.io/projected/3138b353-4c31-4585-82c5-d476bd07f04e-kube-api-access-hbwq2\") pod \"isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.072975 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.072928    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:22:53.105962 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.105923    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:53.379322 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.379197    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"]
Sep 17 22:22:53.382177 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:22:53.382123    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3138b353_4c31_4585_82c5_d476bd07f04e.slice/crio-904fc04bfa840ef1e436e608dd0058d6433e4c8917385910a51d48ae448fa7b3 WatchSource:0}: Error finding container 904fc04bfa840ef1e436e608dd0058d6433e4c8917385910a51d48ae448fa7b3: Status 404 returned error can't find the container with id 904fc04bfa840ef1e436e608dd0058d6433e4c8917385910a51d48ae448fa7b3
Sep 17 22:22:53.944622 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.944588    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" event={"ID":"3138b353-4c31-4585-82c5-d476bd07f04e","Type":"ContainerStarted","Data":"c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a"}
Sep 17 22:22:53.945016 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:53.944655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" event={"ID":"3138b353-4c31-4585-82c5-d476bd07f04e","Type":"ContainerStarted","Data":"904fc04bfa840ef1e436e608dd0058d6433e4c8917385910a51d48ae448fa7b3"}
Sep 17 22:22:57.965436 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:57.965398    2583 generic.go:356] "Generic (PLEG): container finished" podID="3138b353-4c31-4585-82c5-d476bd07f04e" containerID="c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a" exitCode=0
Sep 17 22:22:57.965930 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:57.965475    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" event={"ID":"3138b353-4c31-4585-82c5-d476bd07f04e","Type":"ContainerDied","Data":"c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a"}
Sep 17 22:22:58.972394 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:58.972359    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" event={"ID":"3138b353-4c31-4585-82c5-d476bd07f04e","Type":"ContainerStarted","Data":"190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff"}
Sep 17 22:22:58.972771 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:58.972432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" event={"ID":"3138b353-4c31-4585-82c5-d476bd07f04e","Type":"ContainerStarted","Data":"cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13"}
Sep 17 22:22:58.972771 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:58.972458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" event={"ID":"3138b353-4c31-4585-82c5-d476bd07f04e","Type":"ContainerStarted","Data":"2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9"}
Sep 17 22:22:58.972771 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:58.972730    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:58.972916 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:58.972779    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:58.974255 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:58.974227    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:22:58.996223 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:58.996171    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podStartSLOduration=3.3623924929999998 podStartE2EDuration="6.996154305s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633761812s" isStatefulPod=false podCreationTimestamp="2026-09-17 22:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:22:58.993912182 +0000 UTC m=+1478.077500153" watchObservedRunningTime="2026-09-17 22:22:58.996154305 +0000 UTC m=+1478.079742251"
Sep 17 22:22:59.977066 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:59.977035    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:22:59.977569 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:59.977211    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:22:59.978285 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:22:59.978260    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:23:00.981656 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:00.981611    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:23:00.982106 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:00.982082    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:23:05.985871 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:05.985836    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:23:05.986564 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:05.986534    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:23:05.986904 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:05.986875    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:23:15.987003 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:15.986962    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:23:15.987493 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:15.987444    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:23:23.196429 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:23.196394    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 22:23:23.201384 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:23.201360    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 22:23:23.201541 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:23.201525    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 22:23:23.206075 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:23.206058    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 22:23:25.986783 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:25.986739    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:23:25.987268 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:25.987182    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:23:35.987090 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:35.987002    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:23:35.987598 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:35.987562    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:23:45.986678 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:45.986637    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:23:45.988952 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:45.987109    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:23:55.987301 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:55.987257    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:23:55.987793 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:23:55.987769    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:24:05.986774 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:05.986728    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:24:05.987215 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:05.987182    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:24:09.251510 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:09.251464    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:15.987326 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:15.987292    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:24:15.987878 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:15.987854    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:24:23.833460 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:23.833414    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:28.038385 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:28.038349    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"]
Sep 17 22:24:28.038789 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:28.038638    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" containerID="cri-o://190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff" gracePeriod=30
Sep 17 22:24:28.038789 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:28.038707    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kube-rbac-proxy" containerID="cri-o://cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13" gracePeriod=30
Sep 17 22:24:28.038789 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:28.038622    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" containerID="cri-o://2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9" gracePeriod=30
Sep 17 22:24:28.061477 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:28.061438    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:28.129331 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:28.129301    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:28.382144 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:28.382114    2583 generic.go:356] "Generic (PLEG): container finished" podID="3138b353-4c31-4585-82c5-d476bd07f04e" containerID="cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13" exitCode=2
Sep 17 22:24:28.382308 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:28.382187    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" event={"ID":"3138b353-4c31-4585-82c5-d476bd07f04e","Type":"ContainerDied","Data":"cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13"}
Sep 17 22:24:28.718217 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:28.718062    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:28.815310 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:28.815276    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:24:30.982690 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:30.982644    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 22:24:33.412011 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:33.411981    2583 generic.go:356] "Generic (PLEG): container finished" podID="3138b353-4c31-4585-82c5-d476bd07f04e" containerID="2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9" exitCode=0
Sep 17 22:24:33.412387 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:33.412056    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" event={"ID":"3138b353-4c31-4585-82c5-d476bd07f04e","Type":"ContainerDied","Data":"2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9"}
Sep 17 22:24:35.982109 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:35.982066    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 22:24:35.986417 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:35.986393    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:24:35.988447 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:35.988420    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:24:40.982587 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:40.982540    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 22:24:40.983053 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:40.982701    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:24:44.143406 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:44.143380    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:44.166644 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:44.166613    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:44.176919 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:44.176895    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:44.188950 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:44.188926    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:44.968253 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:44.968228    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:44.980603 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:44.980560    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:44.991425 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:44.991402    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:45.001944 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:45.001922    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:45.833003 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:45.832977    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:45.845781 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:45.845759    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:45.856962 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:45.856942    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:45.871382 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:45.871359    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:45.981810 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:45.981771    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 22:24:45.987069 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:45.987046    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:24:45.988700 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:45.988671    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:24:46.673864 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:46.673839    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:46.684903 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:46.684875    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:46.695367 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:46.695320    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:46.705570 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:46.705540    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:47.504273 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:47.504246    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:47.517350 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:47.517325    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:47.529314 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:47.529292    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:47.541810 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:47.541786    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:48.319665 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:48.319638    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:48.335188 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:48.335160    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:48.348822 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:48.348796    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:48.358314 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:48.358286    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:49.127534 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:49.127503    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:49.139019 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:49.138994    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:49.150764 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:49.150740    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:49.161122 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:49.161099    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:49.938350 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:49.938323    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:49.950324 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:49.950296    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:49.960636 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:49.960615    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:49.971847 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:49.971828    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:50.752509 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:50.752478    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:50.765286 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:50.765262    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:50.776286 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:50.776262    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:50.788050 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:50.788027    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:50.982158 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:50.982096    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 22:24:51.599090 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:51.599059    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:51.610043 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:51.610024    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:51.620078 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:51.620057    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:51.630305 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:51.630288    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:52.418855 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:52.418828    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:52.431073 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:52.431040    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:52.441368 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:52.441347    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:52.452301 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:52.452283    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:53.232054 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:53.232028    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:53.243880 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:53.243856    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:53.254324 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:53.254303    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:53.264264 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:53.264239    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:54.067461 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:54.067436    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:54.079735 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:54.079717    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:54.088779 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:54.088763    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:54.098662 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:54.098645    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:54.883166 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:54.883125    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:54.898116 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:54.898087    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:54.908018 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:54.907994    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:54.918271 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:54.918252    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:55.733203 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:55.733175    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kserve-container/0.log"
Sep 17 22:24:55.745200 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:55.745162    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/kube-rbac-proxy/0.log"
Sep 17 22:24:55.755580 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:55.755562    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/agent/0.log"
Sep 17 22:24:55.766339 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:55.766319    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl_3138b353-4c31-4585-82c5-d476bd07f04e/storage-initializer/0.log"
Sep 17 22:24:55.982061 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:55.982019    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 22:24:55.986782 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:55.986727    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 17 22:24:55.986858 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:55.986844    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:24:55.988403 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:55.988380    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 22:24:55.988510 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:55.988483    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:24:58.191336 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.191307    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:24:58.323725 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.323628    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3138b353-4c31-4585-82c5-d476bd07f04e-proxy-tls\") pod \"3138b353-4c31-4585-82c5-d476bd07f04e\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") "
Sep 17 22:24:58.323725 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.323679    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3138b353-4c31-4585-82c5-d476bd07f04e-kserve-provision-location\") pod \"3138b353-4c31-4585-82c5-d476bd07f04e\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") "
Sep 17 22:24:58.323967 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.323772    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-cabundle-cert\") pod \"3138b353-4c31-4585-82c5-d476bd07f04e\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") "
Sep 17 22:24:58.323967 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.323797    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-kube-rbac-proxy-sar-config\") pod \"3138b353-4c31-4585-82c5-d476bd07f04e\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") "
Sep 17 22:24:58.323967 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.323858    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbwq2\" (UniqueName: \"kubernetes.io/projected/3138b353-4c31-4585-82c5-d476bd07f04e-kube-api-access-hbwq2\") pod \"3138b353-4c31-4585-82c5-d476bd07f04e\" (UID: \"3138b353-4c31-4585-82c5-d476bd07f04e\") "
Sep 17 22:24:58.324112 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.324030    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3138b353-4c31-4585-82c5-d476bd07f04e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3138b353-4c31-4585-82c5-d476bd07f04e" (UID: "3138b353-4c31-4585-82c5-d476bd07f04e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 22:24:58.324540 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.324503    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3138b353-4c31-4585-82c5-d476bd07f04e" (UID: "3138b353-4c31-4585-82c5-d476bd07f04e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:24:58.324649 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.324562    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3138b353-4c31-4585-82c5-d476bd07f04e" (UID: "3138b353-4c31-4585-82c5-d476bd07f04e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 22:24:58.326819 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.326778    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3138b353-4c31-4585-82c5-d476bd07f04e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3138b353-4c31-4585-82c5-d476bd07f04e" (UID: "3138b353-4c31-4585-82c5-d476bd07f04e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 22:24:58.326930 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.326891    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3138b353-4c31-4585-82c5-d476bd07f04e-kube-api-access-hbwq2" (OuterVolumeSpecName: "kube-api-access-hbwq2") pod "3138b353-4c31-4585-82c5-d476bd07f04e" (UID: "3138b353-4c31-4585-82c5-d476bd07f04e"). InnerVolumeSpecName "kube-api-access-hbwq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 22:24:58.425471 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.425429    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbwq2\" (UniqueName: \"kubernetes.io/projected/3138b353-4c31-4585-82c5-d476bd07f04e-kube-api-access-hbwq2\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:24:58.425471 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.425465    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3138b353-4c31-4585-82c5-d476bd07f04e-proxy-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:24:58.425471 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.425474    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3138b353-4c31-4585-82c5-d476bd07f04e-kserve-provision-location\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:24:58.425471 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.425483    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-cabundle-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:24:58.425746 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.425493    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3138b353-4c31-4585-82c5-d476bd07f04e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 17 22:24:58.528524 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.528484    2583 generic.go:356] "Generic (PLEG): container finished" podID="3138b353-4c31-4585-82c5-d476bd07f04e" containerID="190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff" exitCode=0
Sep 17 22:24:58.528698 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.528559    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"
Sep 17 22:24:58.528698 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.528572    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" event={"ID":"3138b353-4c31-4585-82c5-d476bd07f04e","Type":"ContainerDied","Data":"190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff"}
Sep 17 22:24:58.528698 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.528659    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl" event={"ID":"3138b353-4c31-4585-82c5-d476bd07f04e","Type":"ContainerDied","Data":"904fc04bfa840ef1e436e608dd0058d6433e4c8917385910a51d48ae448fa7b3"}
Sep 17 22:24:58.528698 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.528692    2583 scope.go:117] "RemoveContainer" containerID="190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff"
Sep 17 22:24:58.537344 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.537329    2583 scope.go:117] "RemoveContainer" containerID="cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13"
Sep 17 22:24:58.545025 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.545008    2583 scope.go:117] "RemoveContainer" containerID="2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9"
Sep 17 22:24:58.550505 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.550474    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"]
Sep 17 22:24:58.553520 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.553497    2583 scope.go:117] "RemoveContainer" containerID="c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a"
Sep 17 22:24:58.555923 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.555904    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-7c9c6744b4-2tgjl"]
Sep 17 22:24:58.563303 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.563285    2583 scope.go:117] "RemoveContainer" containerID="190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff"
Sep 17 22:24:58.563604 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:24:58.563587    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff\": container with ID starting with 190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff not found: ID does not exist" containerID="190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff"
Sep 17 22:24:58.563650 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.563614    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff"} err="failed to get container status \"190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff\": rpc error: code = NotFound desc = could not find container \"190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff\": container with ID starting with 190dff2f03fa765cbf49a2c35bb53aabcbc3f1ba43835af91be443ebc11fe2ff not found: ID does not exist"
Sep 17 22:24:58.563650 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.563632    2583 scope.go:117] "RemoveContainer" containerID="cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13"
Sep 17 22:24:58.563867 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:24:58.563852    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13\": container with ID starting with cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13 not found: ID does not exist" containerID="cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13"
Sep 17 22:24:58.563910 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.563872    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13"} err="failed to get container status \"cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13\": rpc error: code = NotFound desc = could not find container \"cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13\": container with ID starting with cce0168d1393f4c0f2045be6c4f035833f38e843aa27f62e80b6b5b7df976e13 not found: ID does not exist"
Sep 17 22:24:58.563910 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.563895    2583 scope.go:117] "RemoveContainer" containerID="2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9"
Sep 17 22:24:58.564198 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:24:58.564177    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9\": container with ID starting with 2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9 not found: ID does not exist" containerID="2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9"
Sep 17 22:24:58.564273 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.564205    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9"} err="failed to get container status \"2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9\": rpc error: code = NotFound desc = could not find container \"2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9\": container with ID starting with 2cfd739bf137c334d27307d9f0097c39abedcec44bade6454263aaa53caf24c9 not found: ID does not exist"
Sep 17 22:24:58.564273 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.564228    2583 scope.go:117] "RemoveContainer" containerID="c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a"
Sep 17 22:24:58.564495 ip-10-0-130-119 kubenswrapper[2583]: E0917 22:24:58.564477    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a\": container with ID starting with c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a not found: ID does not exist" containerID="c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a"
Sep 17 22:24:58.564536 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:58.564500    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a"} err="failed to get container status \"c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a\": rpc error: code = NotFound desc = could not find container \"c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a\": container with ID starting with c41e52bddf4ccd8cf8cbc246e2aa18e853835857dd61d76b5dab9583753fc32a not found: ID does not exist"
Sep 17 22:24:59.185839 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:59.185802    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" path="/var/lib/kubelet/pods/3138b353-4c31-4585-82c5-d476bd07f04e/volumes"
Sep 17 22:24:59.728010 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:24:59.727981    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:01.154561 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:01.154526    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:02.691268 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:02.691233    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rffj6_7f93f48f-8169-4a8b-885c-b591f433f601/global-pull-secret-syncer/0.log"
Sep 17 22:25:02.801720 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:02.801689    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log"
Sep 17 22:25:02.881985 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:02.881959    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-119.ec2.internal_ca36c8d3176c4f0a591a991efcdaef22/haproxy/0.log"
Sep 17 22:25:05.273435 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:05.273358    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:05.367374 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:05.367337    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lf995_c1e37671-3221-44a1-b4c0-20f45cec8dcf/konnectivity-agent/0.log" limit=1024
Sep 17 22:25:07.305784 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.305758    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_390aa30d-b0eb-4036-9c3b-1f8520b3e7a9/alertmanager/0.log"
Sep 17 22:25:07.352869 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.352794    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_390aa30d-b0eb-4036-9c3b-1f8520b3e7a9/config-reloader/0.log"
Sep 17 22:25:07.395744 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.395705    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_390aa30d-b0eb-4036-9c3b-1f8520b3e7a9/kube-rbac-proxy-web/0.log"
Sep 17 22:25:07.425173 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.425148    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_390aa30d-b0eb-4036-9c3b-1f8520b3e7a9/kube-rbac-proxy/0.log"
Sep 17 22:25:07.453535 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.453507    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_390aa30d-b0eb-4036-9c3b-1f8520b3e7a9/kube-rbac-proxy-metric/0.log"
Sep 17 22:25:07.477629 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.477605    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_390aa30d-b0eb-4036-9c3b-1f8520b3e7a9/prom-label-proxy/0.log"
Sep 17 22:25:07.508842 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.508822    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_390aa30d-b0eb-4036-9c3b-1f8520b3e7a9/init-config-reloader/0.log"
Sep 17 22:25:07.581885 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.581858    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-j8h4q_d29f1696-1e84-4320-8218-21f95be728d2/kube-state-metrics/0.log"
Sep 17 22:25:07.606998 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.606924    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-j8h4q_d29f1696-1e84-4320-8218-21f95be728d2/kube-rbac-proxy-main/0.log"
Sep 17 22:25:07.634774 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.634745    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-j8h4q_d29f1696-1e84-4320-8218-21f95be728d2/kube-rbac-proxy-self/0.log"
Sep 17 22:25:07.684477 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.684449    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6f5fb6c5b4-9bsjc_e0252fd0-162f-4fc4-a6b2-df0acb473f5f/metrics-server/0.log"
Sep 17 22:25:07.712729 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.712703    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-vc9mj_d1844c7f-fe30-43a7-96d0-1e724a84c003/monitoring-plugin/0.log"
Sep 17 22:25:07.907940 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.907910    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zzlwg_6f2123ef-58e4-400e-8708-622b61aef80b/node-exporter/0.log"
Sep 17 22:25:07.932356 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.932335    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zzlwg_6f2123ef-58e4-400e-8708-622b61aef80b/kube-rbac-proxy/0.log"
Sep 17 22:25:07.954446 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.954423    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zzlwg_6f2123ef-58e4-400e-8708-622b61aef80b/init-textfile/0.log"
Sep 17 22:25:07.990441 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:07.990412    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-hdfbf_1881bf5a-9323-4ce7-bb4f-31c02bb876c0/kube-rbac-proxy-main/0.log"
Sep 17 22:25:08.012961 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:08.012938    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-hdfbf_1881bf5a-9323-4ce7-bb4f-31c02bb876c0/kube-rbac-proxy-self/0.log"
Sep 17 22:25:08.035356 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:08.035332    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-hdfbf_1881bf5a-9323-4ce7-bb4f-31c02bb876c0/openshift-state-metrics/0.log"
Sep 17 22:25:08.347751 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:08.347671    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-9w6xm_6674c479-6e7f-4e5b-9a68-cca01d12d656/prometheus-operator/0.log"
Sep 17 22:25:08.382248 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:08.382211    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-9w6xm_6674c479-6e7f-4e5b-9a68-cca01d12d656/kube-rbac-proxy/0.log"
Sep 17 22:25:08.422236 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:08.422206    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-7fxtf_dfd8e9ad-11d2-4bb2-88d5-d289c3d0de78/prometheus-operator-admission-webhook/0.log"
Sep 17 22:25:09.979864 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:09.979840    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-cvfhk_af9c964d-72a8-4bd8-9823-1eb095dc4463/networking-console-plugin/0.log"
Sep 17 22:25:10.453966 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:10.453937    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/1.log"
Sep 17 22:25:10.458005 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:10.457986    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2kxrz_a5201b1b-c03b-47fc-806d-cce55f2db728/console-operator/2.log"
Sep 17 22:25:10.843857 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:10.843780    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-55f85586b-6mlp9_7a9337c2-28eb-4d24-9247-5dd1b9a838c4/console/0.log"
Sep 17 22:25:11.096869 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.096780    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"]
Sep 17 22:25:11.097346 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.097325    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container"
Sep 17 22:25:11.097437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.097350    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container"
Sep 17 22:25:11.097437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.097364    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent"
Sep 17 22:25:11.097437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.097372    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent"
Sep 17 22:25:11.097437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.097384    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kube-rbac-proxy"
Sep 17 22:25:11.097437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.097394    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kube-rbac-proxy"
Sep 17 22:25:11.097437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.097421    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="storage-initializer"
Sep 17 22:25:11.097437 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.097430    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="storage-initializer"
Sep 17 22:25:11.097770 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.097524    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kserve-container"
Sep 17 22:25:11.097770 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.097541    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="kube-rbac-proxy"
Sep 17 22:25:11.097770 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.097549    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3138b353-4c31-4585-82c5-d476bd07f04e" containerName="agent"
Sep 17 22:25:11.102028 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.102006    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.104637 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.104609    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcd62\"/\"openshift-service-ca.crt\""
Sep 17 22:25:11.104855 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.104838    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fcd62\"/\"default-dockercfg-842mj\""
Sep 17 22:25:11.105392 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.105375    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fcd62\"/\"kube-root-ca.crt\""
Sep 17 22:25:11.111397 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.111377    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"]
Sep 17 22:25:11.245512 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.245473    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-proc\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.245512 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.245513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-sys\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.245736 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.245530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-podres\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.245736 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.245553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgsj6\" (UniqueName: \"kubernetes.io/projected/ac6075df-189f-4c62-b170-af35583c6664-kube-api-access-dgsj6\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.245736 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.245607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-lib-modules\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.346762 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.346729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-lib-modules\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.346938 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.346833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-proc\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.346938 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.346864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-sys\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.346938 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.346887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-podres\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.347079 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.346934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-lib-modules\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.347079 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.346958    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-proc\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.347079 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.346962    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-podres\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.347079 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.346981    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac6075df-189f-4c62-b170-af35583c6664-sys\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.347079 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.346980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgsj6\" (UniqueName: \"kubernetes.io/projected/ac6075df-189f-4c62-b170-af35583c6664-kube-api-access-dgsj6\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.355894 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.355875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgsj6\" (UniqueName: \"kubernetes.io/projected/ac6075df-189f-4c62-b170-af35583c6664-kube-api-access-dgsj6\") pod \"perf-node-gather-daemonset-jh8c4\" (UID: \"ac6075df-189f-4c62-b170-af35583c6664\") " pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.413116 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.413075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:11.683292 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:11.683268    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"]
Sep 17 22:25:11.685848 ip-10-0-130-119 kubenswrapper[2583]: W0917 22:25:11.685821    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podac6075df_189f_4c62_b170_af35583c6664.slice/crio-fc2b94b922c9a1d7aaacaec1074ab256bb9a2e142181836cb3bf27760d0f2a37 WatchSource:0}: Error finding container fc2b94b922c9a1d7aaacaec1074ab256bb9a2e142181836cb3bf27760d0f2a37: Status 404 returned error can't find the container with id fc2b94b922c9a1d7aaacaec1074ab256bb9a2e142181836cb3bf27760d0f2a37
Sep 17 22:25:12.361007 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:12.360982    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2x2z9_370fad22-3b0a-489b-a3f9-f91ff54de074/dns/0.log"
Sep 17 22:25:12.386198 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:12.386172    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2x2z9_370fad22-3b0a-489b-a3f9-f91ff54de074/kube-rbac-proxy/0.log"
Sep 17 22:25:12.564901 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:12.564875    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f4lz4_e9d7824d-7ae5-434d-8ccc-decace1ecd67/dns-node-resolver/0.log"
Sep 17 22:25:12.592723 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:12.592690    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4" event={"ID":"ac6075df-189f-4c62-b170-af35583c6664","Type":"ContainerStarted","Data":"09aaab52ca16a1a82e2f1e37a61ef9e7ae9897d97037a71799f8c42229c232cd"}
Sep 17 22:25:12.592929 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:12.592747    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4" event={"ID":"ac6075df-189f-4c62-b170-af35583c6664","Type":"ContainerStarted","Data":"fc2b94b922c9a1d7aaacaec1074ab256bb9a2e142181836cb3bf27760d0f2a37"}
Sep 17 22:25:12.592929 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:12.592797    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:12.609263 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:12.609212    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4" podStartSLOduration=1.6091958370000001 podStartE2EDuration="1.609195837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 22:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 22:25:12.608487487 +0000 UTC m=+1611.692075459" watchObservedRunningTime="2026-09-17 22:25:12.609195837 +0000 UTC m=+1611.692783785"
Sep 17 22:25:13.083326 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:13.083301    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qfs4z_2acdf951-0d97-4bb7-bbce-cc70bddb8e47/node-ca/0.log"
Sep 17 22:25:13.815098 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:13.815063    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c8d9d759d-mkskv_b7875b9c-d8af-4a71-97ff-b5489c981863/router/0.log"
Sep 17 22:25:14.143495 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:14.143468    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nvdzb_d98da831-8421-4244-819f-08a1f4a46839/serve-healthcheck-canary/0.log"
Sep 17 22:25:14.534638 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:14.534532    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-mhdkx_0a9a9e08-09a2-4062-9b80-d293592d8df2/insights-operator/0.log"
Sep 17 22:25:14.534811 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:14.534785    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-mhdkx_0a9a9e08-09a2-4062-9b80-d293592d8df2/insights-operator/1.log"
Sep 17 22:25:14.696234 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:14.696207    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdc4m_69af59c2-2c6f-46c1-b9d3-ec41a11c14ee/kube-rbac-proxy/0.log"
Sep 17 22:25:14.717600 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:14.717579    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdc4m_69af59c2-2c6f-46c1-b9d3-ec41a11c14ee/exporter/0.log"
Sep 17 22:25:14.738059 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:14.738036    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdc4m_69af59c2-2c6f-46c1-b9d3-ec41a11c14ee/extractor/0.log"
Sep 17 22:25:16.984862 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:16.984831    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-79b679679b-4622b_4d5b680f-dc39-4d5b-9ad2-85890dd122fe/manager/0.log"
Sep 17 22:25:18.608026 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:18.607998    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fcd62/perf-node-gather-daemonset-jh8c4"
Sep 17 22:25:22.870033 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:22.869936    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2n6n9_2c8fb3d3-7b60-4cc8-92d7-067809b20b92/kube-multus/0.log"
Sep 17 22:25:23.238930 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:23.238898    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlb9z_6df8b67b-5e30-4cbb-8495-6f78f2524e13/kube-multus-additional-cni-plugins/0.log"
Sep 17 22:25:23.259315 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:23.259292    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlb9z_6df8b67b-5e30-4cbb-8495-6f78f2524e13/egress-router-binary-copy/0.log"
Sep 17 22:25:23.279332 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:23.279310    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlb9z_6df8b67b-5e30-4cbb-8495-6f78f2524e13/cni-plugins/0.log"
Sep 17 22:25:23.299681 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:23.299644    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlb9z_6df8b67b-5e30-4cbb-8495-6f78f2524e13/bond-cni-plugin/0.log"
Sep 17 22:25:23.320412 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:23.320386    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlb9z_6df8b67b-5e30-4cbb-8495-6f78f2524e13/routeoverride-cni/0.log"
Sep 17 22:25:23.341083 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:23.341060    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlb9z_6df8b67b-5e30-4cbb-8495-6f78f2524e13/whereabouts-cni-bincopy/0.log"
Sep 17 22:25:23.361232 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:23.361205    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlb9z_6df8b67b-5e30-4cbb-8495-6f78f2524e13/whereabouts-cni/0.log"
Sep 17 22:25:23.466769 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:23.466743    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n64gh_6cc2ec45-0451-46da-aa90-214fd10d39bd/network-metrics-daemon/0.log"
Sep 17 22:25:23.488052 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:23.488028    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n64gh_6cc2ec45-0451-46da-aa90-214fd10d39bd/kube-rbac-proxy/0.log"
Sep 17 22:25:24.750693 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:24.750663    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-controller/0.log"
Sep 17 22:25:24.772926 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:24.772901    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/1.log"
Sep 17 22:25:24.772926 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:24.772922    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovn-acl-logging/0.log"
Sep 17 22:25:24.793552 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:24.793528    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/kube-rbac-proxy-node/0.log"
Sep 17 22:25:24.814589 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:24.814561    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 22:25:24.834870 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:24.834843    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/northd/0.log"
Sep 17 22:25:24.857553 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:24.857532    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/nbdb/0.log"
Sep 17 22:25:24.881234 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:24.881211    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/sbdb/0.log"
Sep 17 22:25:24.924799 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:24.924773    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vpmct_628981f9-9f9e-4bb1-9eb1-fd1b0241989f/ovnkube-controller/0.log"
Sep 17 22:25:25.935652 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:25.935624    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-q8xvj_032d58e2-2e31-40c8-87cc-a4d5939db410/network-check-target-container/0.log"
Sep 17 22:25:26.905697 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:26.905668    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qqc5b_a58f9aab-8cec-4b05-979e-7831f7505070/iptables-alerter/0.log"
Sep 17 22:25:27.639169 ip-10-0-130-119 kubenswrapper[2583]: I0917 22:25:27.639125    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-wcphq_a921b3aa-f930-4ac3-9cb4-9af843a381df/tuned/0.log"