Sep 17 19:06:15.542495 ip-10-0-140-225 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 19:06:18.884744 ip-10-0-140-225 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:18.884744 ip-10-0-140-225 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 19:06:18.884744 ip-10-0-140-225 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:18.884744 ip-10-0-140-225 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 19:06:18.884744 ip-10-0-140-225 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:06:18.885585 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.885261    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 19:06:18.890035 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890004    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:18.890035 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890026    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:18.890035 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890031    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:18.890035 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890036    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:18.890035 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890039    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:18.890035 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890044    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890048    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890052    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890056    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890059    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890064    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890068    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890072    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890075    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890079    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890082    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890086    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890090    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890094    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890097    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890101    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890104    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890108    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890112    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890115    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:18.890417 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890119    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890123    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890127    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890133    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890156    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890167    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890177    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890180    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890184    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890188    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890192    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890196    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890200    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890204    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890209    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890214    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890220    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890225    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890229    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890233    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:18.891251 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890237    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890241    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890246    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890250    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890255    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890259    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890264    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890268    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890272    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890277    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890281    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890285    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890289    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890293    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890297    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890301    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890306    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890310    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890315    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890320    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:18.892112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890324    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890328    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890333    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890337    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890341    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890345    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890350    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890355    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890359    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890367    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890374    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890380    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890384    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890389    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890393    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890398    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890402    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890406    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890422    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:18.892955 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890427    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890430    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890434    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890439    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890443    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890448    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890452    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890456    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890460    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890464    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890468    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890472    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890477    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890481    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.890485    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891741    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891751    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891756    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891761    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891767    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891771    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:18.893862 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891776    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891780    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891787    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891794    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891801    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891806    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891811    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891815    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891820    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891824    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891829    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891844    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891849    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891853    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891857    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891862    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891866    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891871    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891875    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:18.894772 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891879    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891884    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891888    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891892    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891898    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891902    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891906    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891910    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891914    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891919    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891923    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891928    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891933    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891937    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891941    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891945    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891949    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891954    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891958    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891963    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:18.895444 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891968    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891974    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891978    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891982    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.891987    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892002    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892007    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892012    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892017    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892021    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892026    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892030    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892034    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892038    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892043    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892047    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892051    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892055    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892059    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:18.895933 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892064    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892068    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892072    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892076    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892080    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892086    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892090    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892094    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892098    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892103    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892107    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892110    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892114    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892118    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892122    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892126    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892130    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892152    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892157    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:18.896512 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892170    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892174    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892178    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892183    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892187    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892191    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892196    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892200    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892204    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892209    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892214    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892218    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892223    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892227    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892231    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892235    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892918    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892926    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892931    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:18.897009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892938    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892943    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892948    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892953    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892957    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892961    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892965    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892969    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892972    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892976    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892981    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892985    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.892989    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893001    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893006    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893010    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893015    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893019    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893023    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893027    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:18.897591 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893031    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893036    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893041    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893045    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893049    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893053    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893058    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893062    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893066    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893070    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893074    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893078    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893082    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893087    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893091    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893095    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893099    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893103    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893107    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893111    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:18.898256 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893116    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893121    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893128    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893133    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893157    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893161    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893174    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893178    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893183    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893186    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893191    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893195    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893199    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893207    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893211    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893215    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893219    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893223    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893228    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893232    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:18.898740 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893236    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893240    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893245    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893249    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893253    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893257    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893261    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893265    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893270    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893274    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893279    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893283    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893298    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893304    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893309    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893313    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893317    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893321    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893325    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:18.899354 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893337    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893342    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893346    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893349    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893353    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893357    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893364    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893368    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893372    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893376    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893381    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893385    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893390    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893394    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893399    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893404    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.893408    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893519    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893571    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893585    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893592    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 19:06:18.899808 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893599    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893604    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893611    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893618    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893624    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893628    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893634    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893639    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893644    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893649    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893653    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893658    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893673    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893680    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893691    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893696    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893701    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893708    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893713    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893719    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893724    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893729    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893734    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893739    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 19:06:18.900329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893744    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893749    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893754    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893759    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893765    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893770    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893774    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893779    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893784    2567 flags.go:64] FLAG: --enable-server="true"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893789    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893799    2567 flags.go:64] FLAG: --event-burst="100"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893803    2567 flags.go:64] FLAG: --event-qps="50"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893808    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893813    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893817    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893823    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893828    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893833    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893837    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893842    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893847    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893861    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893866    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893870    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893875    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 19:06:18.900933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893881    2567 flags.go:64] FLAG: --feature-gates=""
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893888    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893893    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893898    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893904    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893908    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893913    2567 flags.go:64] FLAG: --help="false"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893918    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-140-225.ec2.internal"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893923    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893928    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893932    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893938    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893943    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893948    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893952    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893957    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893962    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893966    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893972    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893976    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893981    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893985    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893990    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.893995    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 19:06:18.901567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894000    2567 flags.go:64] FLAG: --lock-file=""
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894004    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894009    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894014    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894028    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894041    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894046    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894050    2567 flags.go:64] FLAG: --logging-format="text"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894057    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894062    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894067    2567 flags.go:64] FLAG: --manifest-url=""
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894072    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894079    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894084    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894090    2567 flags.go:64] FLAG: --max-pods="110"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894095    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894100    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894104    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894109    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894114    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894119    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894124    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894159    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894175    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894180    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 19:06:18.902236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894184    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894189    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894198    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894202    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894207    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894212    2567 flags.go:64] FLAG: --port="10250"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894216    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894221    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03f1bedbd68843f9b"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894226    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894231    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894236    2567 flags.go:64] FLAG: --register-node="true"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894242    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894249    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894265    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894270    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894275    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894283    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894296    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894301    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894306    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894311    2567 flags.go:64] FLAG: --runonce="false"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894316    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894321    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894325    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894330    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 19:06:18.902850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894335    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894340    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894345    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894350    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894354    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894359    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894364    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894368    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894373    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894378    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894382    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894392    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894397    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894401    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894412    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894416    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894421    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894425    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894432    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894437    2567 flags.go:64] FLAG: --v="2"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894443    2567 flags.go:64] FLAG: --version="false"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894461    2567 flags.go:64] FLAG: --vmodule=""
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894468    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.894475    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894682    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:18.903483 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894688    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894694    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894701    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894708    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894713    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894717    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894723    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894730    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894734    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894738    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894743    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894747    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894751    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894756    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894760    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894764    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894769    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894773    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894777    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:18.904085 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894781    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894785    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894789    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894793    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894797    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894802    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894808    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894812    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894817    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894821    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894826    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894832    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894837    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894841    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894845    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894850    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894854    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894858    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894862    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894867    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:18.904623 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894871    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894875    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894879    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894883    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894887    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894892    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894896    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894900    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894904    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894909    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894913    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894917    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894921    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894925    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894930    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894934    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894938    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894942    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894948    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894952    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:18.905114 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894956    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894961    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894965    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894972    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894977    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894981    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894985    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894990    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.894994    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895000    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895005    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895010    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895014    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895018    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895023    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895027    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895032    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895036    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895040    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:18.905617 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895044    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895048    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895052    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895056    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895061    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895065    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895069    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895073    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895077    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895082    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895086    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895091    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895095    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895100    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895104    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895108    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895114    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895120    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895125    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:18.906083 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.895129    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.896025    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.903519    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.903535    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.903556    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.903561    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903618    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903623    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903626    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903630    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903632    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903635    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903638    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903641    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:18.906577 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903644    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903646    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903649    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903652    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903655    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903659    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903663    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903666    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903669    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903671    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903674    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903676    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903679    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903682    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903685    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903687    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903690    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903693    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903696    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:18.906945 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903698    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903702    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903705    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903708    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903710    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903713    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903716    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903719    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903721    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903724    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903727    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903729    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903732    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903735    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903738    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903741    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903744    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903746    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903749    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903751    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:18.907418 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903754    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903757    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903759    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903761    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903764    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903766    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903769    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903772    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903774    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903777    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903782    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903786    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903789    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903792    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903795    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903798    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903801    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903803    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903806    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903809    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:18.907942 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903811    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903814    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903817    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903819    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903821    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903824    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903827    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903829    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903832    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903834    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903837    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903840    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903842    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903846    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903850    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903853    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903856    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903859    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903862    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:18.908449 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903864    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903867    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903870    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903872    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903880    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903882    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903885    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903888    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903891    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903894    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903896    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903899    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.903902    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.903907    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904021    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:06:18.908916 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904025    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904027    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904031    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904034    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904037    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904040    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904043    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904046    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904049    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904052    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904055    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904058    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904061    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904064    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904066    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904069    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904072    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904075    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904077    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904080    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:06:18.909381 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904083    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904086    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904089    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904092    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904095    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904098    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904101    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904103    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904106    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904109    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904111    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904114    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904116    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904119    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904121    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904124    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904126    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904130    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904134    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:06:18.909855 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904152    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904155    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904158    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904160    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904163    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904166    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904169    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904171    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904174    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904177    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904180    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904183    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904185    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904188    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904191    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904195    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904199    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904203    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904205    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:06:18.910338 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904208    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904211    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904214    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904216    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904219    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904221    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904224    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904227    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904229    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904232    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904235    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904239    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904242    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904245    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904248    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904250    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904253    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904256    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904259    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904262    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:06:18.910793 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904264    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904267    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904270    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904273    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904276    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904278    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904281    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904284    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904287    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904289    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904292    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904295    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904297    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904300    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904302    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904305    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904307    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904309    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904312    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:06:18.911296 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:18.904315    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:06:18.911757 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.904319    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:06:18.911757 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.904417    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 19:06:18.911757 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.907215    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 19:06:18.911757 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.908061    2567 server.go:1019] "Starting client certificate rotation"
Sep 17 19:06:18.911757 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.908167    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:18.911757 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.908194    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:06:18.911757 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.910265    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:18.913114 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.913000    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:06:18.926950 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.926932    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 17 19:06:18.928781 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.928763    2567 log.go:25] "Validated CRI v1 image API"
Sep 17 19:06:18.930129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.930113    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 19:06:18.932316 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.932296    2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d132b22c-1c58-4676-a45e-d8e2a8a9ad89:/dev/nvme0n1p3 e1a89d6d-164a-4260-a4e5-eaf761d57a6a:/dev/nvme0n1p4]
Sep 17 19:06:18.932394 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.932315    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 19:06:18.936615 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.936497    2567 manager.go:216] Machine: {Timestamp:2026-09-17 19:06:18.935996625 +0000 UTC m=+1.862215038 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102616 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c0e96c68f64abab48538e5997db0e SystemUUID:ec2c0e96-c68f-64ab-ab48-538e5997db0e BootID:ff2e65b8-8af6-437d-87e3-560e3103211b Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:83:b2:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:83:b2:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:7c:35:2f:33:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 19:06:18.936615 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.936606    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 19:06:18.936742 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.936715    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 19:06:18.938286 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.938258    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 19:06:18.938460 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.938289    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-225.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 19:06:18.939598 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.939588    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 19:06:18.939638 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.939601    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 19:06:18.939638 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.939609    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:18.939638 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.939625    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:06:18.940949 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.940939    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 19:06:18.944955 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.944941    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:18.945070 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.945060    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 19:06:18.945108 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.945089    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 19:06:18.945108 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.945101    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 19:06:18.945177 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.945117    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 19:06:18.945177 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.945129    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 17 19:06:18.945177 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.945149    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 19:06:18.945177 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.945154    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:18.945569 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.945556    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:18.945617 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.945581    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:06:18.947580 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.947561    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 19:06:18.951336 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.951320    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 19:06:18.951406 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.951351    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 19:06:18.952580 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952568    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 19:06:18.952630 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952586    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 19:06:18.952630 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952592    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 19:06:18.952630 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952597    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 19:06:18.952630 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952606    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 19:06:18.952630 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952611    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 19:06:18.952630 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952624    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 19:06:18.952630 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952629    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 19:06:18.952807 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952636    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 19:06:18.952807 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952641    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 19:06:18.952807 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952656    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 19:06:18.952807 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952661    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 19:06:18.952807 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952690    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 19:06:18.952807 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.952696    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 19:06:18.955048 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:18.955010    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-225.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 19:06:18.955152 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:18.955057    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 19:06:18.959423 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.959405    2567 server.go:1268] "Started kubelet"
Sep 17 19:06:18.959513 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.959458    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 19:06:18.959597 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.959547    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 19:06:18.959688 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.959617    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 19:06:18.960243 ip-10-0-140-225 systemd[1]: Started Kubernetes Kubelet.
Sep 17 19:06:18.960659 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.960645    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 19:06:18.962011 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.961992    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-225.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 19:06:18.963461 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.963442    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 19:06:18.964038 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:18.964017    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 19:06:18.969206 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.969190    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 19:06:18.969275 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.969221    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:18.969337 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:18.969268    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-225.ec2.internal\" not found"
Sep 17 19:06:18.969767 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.969747    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 19:06:18.969767 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.969747    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 19:06:18.969898 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.969778    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 19:06:18.969898 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.969866    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 19:06:18.969898 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.969876    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 19:06:18.970109 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.970093    2567 factory.go:55] Registering systemd factory
Sep 17 19:06:18.970192 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.970117    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 17 19:06:18.970363 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.970344    2567 factory.go:153] Registering CRI-O factory
Sep 17 19:06:18.970363 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.970363    2567 factory.go:223] Registration of the crio container factory successfully
Sep 17 19:06:18.970510 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.970433    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 19:06:18.970510 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.970453    2567 factory.go:103] Registering Raw factory
Sep 17 19:06:18.970510 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.970483    2567 manager.go:1196] Started watching for new ooms in manager
Sep 17 19:06:18.970865 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.970852    2567 manager.go:349] Starting recovery of all containers
Sep 17 19:06:18.972221 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:18.972196    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-225.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 19:06:18.972314 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:18.972292    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 19:06:18.972979 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.972956    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x64fh"
Sep 17 19:06:18.982273 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.982246    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x64fh"
Sep 17 19:06:18.982760 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:18.982738    2567 manager.go:354] Recovery completed
Sep 17 19:06:19.003338 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.003316    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:19.005593 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.005577    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:19.005674 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.005605    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:19.005674 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.005615    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:19.006211 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.006194    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 19:06:19.006211 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.006206    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:19.006371 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.006210    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 19:06:19.006371 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.006230    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:06:19.006934 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.006912    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:19.007009 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.006940    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:19.007009 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.006950    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:19.008332 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.008319    2567 policy_none.go:49] "None policy: Start"
Sep 17 19:06:19.008403 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.008339    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 19:06:19.008403 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.008352    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 19:06:19.010363 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.010351    2567 policy_none.go:47] "Start"
Sep 17 19:06:19.055688 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.051759    2567 manager.go:337] "Starting Device Plugin manager"
Sep 17 19:06:19.055688 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.051798    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 19:06:19.055688 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.051810    2567 server.go:87] "Starting device plugin registration server"
Sep 17 19:06:19.055688 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.052088    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 19:06:19.055688 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.052110    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 19:06:19.055688 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.052219    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 19:06:19.055688 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.052324    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 19:06:19.055688 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.052332    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 19:06:19.055688 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.052865    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 19:06:19.055688 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.052924    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-225.ec2.internal\" not found"
Sep 17 19:06:19.075349 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.075318    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 19:06:19.076761 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.076737    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 19:06:19.076875 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.076788    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 19:06:19.076875 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.076812    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 19:06:19.076875 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.076850    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 19:06:19.079285 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.079264    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:19.153017 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.152901    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:19.153944 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.153929    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:19.154029 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.153958    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:19.154029 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.153969    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:19.154029 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.153993    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.163378 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.163358    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.163443 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.163381    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-225.ec2.internal\": node \"ip-10-0-140-225.ec2.internal\" not found"
Sep 17 19:06:19.177103 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.177072    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-225.ec2.internal"]
Sep 17 19:06:19.177221 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.177156    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:19.178538 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.178522    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:19.178626 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.178553    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:19.178626 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.178566    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:19.180920 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.180906    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:19.181039 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.181026    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.181083 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.181058    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:19.181629 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.181609    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:19.181736 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.181638    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:19.181736 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.181648    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:19.181736 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.181616    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:19.181736 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.181723    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:19.181926 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.181740    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:19.184047 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.184031    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.184156 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.184057    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:06:19.184770 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.184755    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:06:19.184850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.184785    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:06:19.184850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.184799    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:06:19.202022 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.201996    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-225.ec2.internal\" not found"
Sep 17 19:06:19.206888 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.206869    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-225.ec2.internal\" not found" node="ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.211260 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.211242    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-225.ec2.internal\" not found" node="ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.271472 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.271442    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3fa8c6c6ae763d57f9ebb8f3758a86e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal\" (UID: \"e3fa8c6c6ae763d57f9ebb8f3758a86e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.271472 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.271474    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3fa8c6c6ae763d57f9ebb8f3758a86e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal\" (UID: \"e3fa8c6c6ae763d57f9ebb8f3758a86e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.271674 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.271494    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0937b68b17a7b35dfeea96e89aad3eb3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-225.ec2.internal\" (UID: \"0937b68b17a7b35dfeea96e89aad3eb3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.302410 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.302368    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-225.ec2.internal\" not found"
Sep 17 19:06:19.371710 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.371682    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3fa8c6c6ae763d57f9ebb8f3758a86e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal\" (UID: \"e3fa8c6c6ae763d57f9ebb8f3758a86e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.371801 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.371719    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0937b68b17a7b35dfeea96e89aad3eb3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-225.ec2.internal\" (UID: \"0937b68b17a7b35dfeea96e89aad3eb3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.371801 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.371736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3fa8c6c6ae763d57f9ebb8f3758a86e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal\" (UID: \"e3fa8c6c6ae763d57f9ebb8f3758a86e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.371801 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.371790    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3fa8c6c6ae763d57f9ebb8f3758a86e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal\" (UID: \"e3fa8c6c6ae763d57f9ebb8f3758a86e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.371958 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.371798    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3fa8c6c6ae763d57f9ebb8f3758a86e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal\" (UID: \"e3fa8c6c6ae763d57f9ebb8f3758a86e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.371958 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.371819    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0937b68b17a7b35dfeea96e89aad3eb3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-225.ec2.internal\" (UID: \"0937b68b17a7b35dfeea96e89aad3eb3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.402829 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.402784    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-225.ec2.internal\" not found"
Sep 17 19:06:19.503655 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.503592    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-225.ec2.internal\" not found"
Sep 17 19:06:19.510757 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.510733    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.515069 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.515040    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.604394 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.604351    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-225.ec2.internal\" not found"
Sep 17 19:06:19.704954 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.704925    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-225.ec2.internal\" not found"
Sep 17 19:06:19.805468 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.805407    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-225.ec2.internal\" not found"
Sep 17 19:06:19.847448 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.847411    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:19.869757 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.869718    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.880268 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.880244    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:06:19.881727 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.881712    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal"
Sep 17 19:06:19.892664 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.892642    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:06:19.907856 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.907838    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 19:06:19.907980 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.907959    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 19:06:19.908038 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.908005    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 19:06:19.946192 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.946154    2567 apiserver.go:52] "Watching apiserver"
Sep 17 19:06:19.966150 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.966123    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 19:06:19.968561 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.968527    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-g5mvq","openshift-multus/multus-additional-cni-plugins-fwmhb","openshift-multus/network-metrics-daemon-8gl6r","openshift-network-diagnostics/network-check-target-mshnf","kube-system/kube-apiserver-proxy-ip-10-0-140-225.ec2.internal","openshift-dns/node-resolver-xhmnn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal","openshift-multus/multus-c5cgz","openshift-network-operator/iptables-alerter-7j5gq","openshift-ovn-kubernetes/ovnkube-node-cw78b","kube-system/global-pull-secret-syncer-wx589","kube-system/konnectivity-agent-2z92j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc","openshift-cluster-node-tuning-operator/tuned-gvnfj"]
Sep 17 19:06:19.969635 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.969616    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 19:06:19.971165 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.971117    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g5mvq"
Sep 17 19:06:19.973448 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.973425    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:19.975078 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.975047    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 19:06:19.975078 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.975060    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8c8d54f-7756-4625-94b7-806ff2e8db82-host\") pod \"node-ca-g5mvq\" (UID: \"f8c8d54f-7756-4625-94b7-806ff2e8db82\") " pod="openshift-image-registry/node-ca-g5mvq"
Sep 17 19:06:19.975249 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.975096    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8c8d54f-7756-4625-94b7-806ff2e8db82-serviceca\") pod \"node-ca-g5mvq\" (UID: \"f8c8d54f-7756-4625-94b7-806ff2e8db82\") " pod="openshift-image-registry/node-ca-g5mvq"
Sep 17 19:06:19.975249 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.975123    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cvgt\" (UniqueName: \"kubernetes.io/projected/f8c8d54f-7756-4625-94b7-806ff2e8db82-kube-api-access-2cvgt\") pod \"node-ca-g5mvq\" (UID: \"f8c8d54f-7756-4625-94b7-806ff2e8db82\") " pod="openshift-image-registry/node-ca-g5mvq"
Sep 17 19:06:19.975314 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.975258    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 19:06:19.975527 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.975515    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:19.975588 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.975571    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:19.977435 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.977414    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 19:06:19.977980 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.977961    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 19:06:19.978054 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.978044    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 19:06:19.978099 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.978083    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 19:06:19.978099 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.978089    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mx6q8\""
Sep 17 19:06:19.978245 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.978114    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 19:06:19.978245 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.978082    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wzg6z\""
Sep 17 19:06:19.978245 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.978157    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 19:06:19.978403 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.978388    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:19.978467 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.978449    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:19.980519 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.980493    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xhmnn"
Sep 17 19:06:19.982731 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.982714    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:19.983881 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.983865    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 19:06:19.984187 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.984130    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h6r4v\""
Sep 17 19:06:19.984491 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.984472    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 19:06:19.984990 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.984976    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7j5gq"
Sep 17 19:06:19.986250 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.986232    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:19.987118 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.987102    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qnkzk\""
Sep 17 19:06:19.987400 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.987386    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:19.988530 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.988511    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 19:06:19.989680 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.989661    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:19.989809 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:19.989731    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:19.990112 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.990096    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lmqvs\""
Sep 17 19:06:19.990257 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.990191    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:19.990257 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.990221    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 19:06:19.990379 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.990296    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 19:06:19.990476 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.990456    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 19:06:19.990562 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.990548    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 19:06:19.991518 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.991498    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-76vv6\""
Sep 17 19:06:19.991602 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.991551    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 19:06:19.991602 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.991591    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 19:06:19.991700 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.991591    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:19.991700 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.991617    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:06:19.992574 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.992555    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:19.994651 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.994636    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 19:06:19.995465 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.995309    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:19.995528 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.995516    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 19:06:19.995577 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.995549    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2klb7\""
Sep 17 19:06:19.995812 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.995799    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 19:06:19.997722 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.997699    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 19:06:19.997813 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.997795    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:19.999676 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.998852    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 19:06:19.999676 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.999132    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 19:06:19.999676 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:19.999433    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-csmsx\""
Sep 17 19:06:20.001464 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.001033    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 19:01:18 +0000 UTC" deadline="2028-04-19 14:24:28.309446484 +0000 UTC"
Sep 17 19:06:20.001464 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.001093    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13915h18m8.308357696s"
Sep 17 19:06:20.002428 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.002411    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:20.002526 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.002504    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:20.002600 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.002552    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dcc46\""
Sep 17 19:06:20.043194 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.043166    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sq6fk"
Sep 17 19:06:20.050554 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.050523    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sq6fk"
Sep 17 19:06:20.071258 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.071237    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 19:06:20.075564 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-etc-kubernetes\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.075673 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075574    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-systemd-units\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.075673 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075594    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-slash\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.075673 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075609    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvq5v\" (UniqueName: \"kubernetes.io/projected/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-kube-api-access-xvq5v\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.075673 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075625    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:20.075673 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075646    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-registration-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.075673 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075669    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-modprobe-d\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.075925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075707    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-kubernetes\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.075925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075729    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-tuned\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.075925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075758    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-system-cni-dir\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.075925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075777    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-cni-binary-copy\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.075925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075790    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-run-systemd\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.075925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075826    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-sysctl-d\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.075925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075853    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.075925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075878    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebba4c-75f7-4b76-a077-c9bbbfd109e0-konnectivity-ca\") pod \"konnectivity-agent-2z92j\" (UID: \"e6ebba4c-75f7-4b76-a077-c9bbbfd109e0\") " pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:20.075925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075910    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-hostroot\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.076350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-run-ovn-kubernetes\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.076350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.075962    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.076350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076028    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-lib-modules\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.076350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076085    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4da49179-a694-49ea-a836-7ce5e0fbfb13-dbus\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:20.076350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076125    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-socket-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.076350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076178    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-systemd\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.076350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076214    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-kubelet\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.076350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076251    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-run-ovn\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.076350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076274    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-ovnkube-config\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.076350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076296    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-device-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.076350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076335    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-etc-selinux\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076358    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggwtv\" (UniqueName: \"kubernetes.io/projected/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-kube-api-access-ggwtv\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076397    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-socket-dir-parent\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076429    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h75rh\" (UniqueName: \"kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh\") pod \"network-check-target-mshnf\" (UID: \"f663602d-7308-4423-880f-327ce634f45b\") " pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076464    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-sysconfig\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076497    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-host\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076529    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqngb\" (UniqueName: \"kubernetes.io/projected/223bc150-3bae-4e3a-a45e-69865f53a23d-kube-api-access-qqngb\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076553    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbadfc18-65c1-4256-b7da-f153a1a188e9-host-slash\") pod \"iptables-alerter-7j5gq\" (UID: \"fbadfc18-65c1-4256-b7da-f153a1a188e9\") " pod="openshift-network-operator/iptables-alerter-7j5gq"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076589    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7km4\" (UniqueName: \"kubernetes.io/projected/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-kube-api-access-j7km4\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076613    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076638    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076681    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cvgt\" (UniqueName: \"kubernetes.io/projected/f8c8d54f-7756-4625-94b7-806ff2e8db82-kube-api-access-2cvgt\") pod \"node-ca-g5mvq\" (UID: \"f8c8d54f-7756-4625-94b7-806ff2e8db82\") " pod="openshift-image-registry/node-ca-g5mvq"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076705    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5744cdd7-665f-41f2-96b7-150f693100a2-tmp-dir\") pod \"node-resolver-xhmnn\" (UID: \"5744cdd7-665f-41f2-96b7-150f693100a2\") " pod="openshift-dns/node-resolver-xhmnn"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076754    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-conf-dir\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-etc-openvswitch\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076836    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-tmp\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.076962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076859    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5744cdd7-665f-41f2-96b7-150f693100a2-hosts-file\") pod \"node-resolver-xhmnn\" (UID: \"5744cdd7-665f-41f2-96b7-150f693100a2\") " pod="openshift-dns/node-resolver-xhmnn"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076879    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-system-cni-dir\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076902    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-var-lib-cni-bin\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076962    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-var-lib-kubelet\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.076992    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgsl8\" (UniqueName: \"kubernetes.io/projected/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-kube-api-access-lgsl8\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077036    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-var-lib-openvswitch\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077059    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-run-openvswitch\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077094    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-ovn-node-metrics-cert\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077156    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-cni-netd\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077189    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077215    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-var-lib-kubelet\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077242    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077264    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-cni-bin\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077279    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-ovnkube-script-lib\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077297    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm8xq\" (UniqueName: \"kubernetes.io/projected/fbadfc18-65c1-4256-b7da-f153a1a188e9-kube-api-access-hm8xq\") pod \"iptables-alerter-7j5gq\" (UID: \"fbadfc18-65c1-4256-b7da-f153a1a188e9\") " pod="openshift-network-operator/iptables-alerter-7j5gq"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077325    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-run\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.077658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077362    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5nb\" (UniqueName: \"kubernetes.io/projected/5744cdd7-665f-41f2-96b7-150f693100a2-kube-api-access-hp5nb\") pod \"node-resolver-xhmnn\" (UID: \"5744cdd7-665f-41f2-96b7-150f693100a2\") " pod="openshift-dns/node-resolver-xhmnn"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077406    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-sys\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077450    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077483    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-daemon-config\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077506    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-run-netns\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077550    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-env-overrides\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077572    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-sysctl-conf\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077599    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8c8d54f-7756-4625-94b7-806ff2e8db82-serviceca\") pod \"node-ca-g5mvq\" (UID: \"f8c8d54f-7756-4625-94b7-806ff2e8db82\") " pod="openshift-image-registry/node-ca-g5mvq"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077620    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-cni-dir\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077678    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-run-netns\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077859    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fbadfc18-65c1-4256-b7da-f153a1a188e9-iptables-alerter-script\") pod \"iptables-alerter-7j5gq\" (UID: \"fbadfc18-65c1-4256-b7da-f153a1a188e9\") " pod="openshift-network-operator/iptables-alerter-7j5gq"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077901    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-sys-fs\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.077965    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-run-k8s-cni-cncf-io\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078004    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-run-multus-certs\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078036    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-node-log\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078108    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f8c8d54f-7756-4625-94b7-806ff2e8db82-serviceca\") pod \"node-ca-g5mvq\" (UID: \"f8c8d54f-7756-4625-94b7-806ff2e8db82\") " pod="openshift-image-registry/node-ca-g5mvq"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078123    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-log-socket\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.078369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078178    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4da49179-a694-49ea-a836-7ce5e0fbfb13-kubelet-config\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:20.079161 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078205    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rk6h\" (UniqueName: \"kubernetes.io/projected/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-kube-api-access-4rk6h\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.079161 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-cnibin\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.079161 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078257    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6ebba4c-75f7-4b76-a077-c9bbbfd109e0-agent-certs\") pod \"konnectivity-agent-2z92j\" (UID: \"e6ebba4c-75f7-4b76-a077-c9bbbfd109e0\") " pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:20.079161 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8c8d54f-7756-4625-94b7-806ff2e8db82-host\") pod \"node-ca-g5mvq\" (UID: \"f8c8d54f-7756-4625-94b7-806ff2e8db82\") " pod="openshift-image-registry/node-ca-g5mvq"
Sep 17 19:06:20.079161 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078318    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-cnibin\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.079161 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078359    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f8c8d54f-7756-4625-94b7-806ff2e8db82-host\") pod \"node-ca-g5mvq\" (UID: \"f8c8d54f-7756-4625-94b7-806ff2e8db82\") " pod="openshift-image-registry/node-ca-g5mvq"
Sep 17 19:06:20.079161 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.078409    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-var-lib-cni-multus\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.092542 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.092523    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 19:06:20.096078 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.096057    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cvgt\" (UniqueName: \"kubernetes.io/projected/f8c8d54f-7756-4625-94b7-806ff2e8db82-kube-api-access-2cvgt\") pod \"node-ca-g5mvq\" (UID: \"f8c8d54f-7756-4625-94b7-806ff2e8db82\") " pod="openshift-image-registry/node-ca-g5mvq"
Sep 17 19:06:20.178654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178619    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-cni-bin\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.178654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178657    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-ovnkube-script-lib\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.178819 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178674    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm8xq\" (UniqueName: \"kubernetes.io/projected/fbadfc18-65c1-4256-b7da-f153a1a188e9-kube-api-access-hm8xq\") pod \"iptables-alerter-7j5gq\" (UID: \"fbadfc18-65c1-4256-b7da-f153a1a188e9\") " pod="openshift-network-operator/iptables-alerter-7j5gq"
Sep 17 19:06:20.178819 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178729    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-cni-bin\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.178819 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178776    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-run\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.178819 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178803    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5nb\" (UniqueName: \"kubernetes.io/projected/5744cdd7-665f-41f2-96b7-150f693100a2-kube-api-access-hp5nb\") pod \"node-resolver-xhmnn\" (UID: \"5744cdd7-665f-41f2-96b7-150f693100a2\") " pod="openshift-dns/node-resolver-xhmnn"
Sep 17 19:06:20.178967 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178904    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-sys\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.178967 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178917    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-run\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.178967 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178933    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.178967 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178959    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-daemon-config\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.179167 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178967    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-sys\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.179167 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.178993    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-run-netns\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.179167 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179016    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-env-overrides\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.179167 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179050    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-sysctl-conf\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.179167 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179095    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-cni-dir\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.179167 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179108    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-run-netns\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179167    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-run-netns\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179122    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-run-netns\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179212    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fbadfc18-65c1-4256-b7da-f153a1a188e9-iptables-alerter-script\") pod \"iptables-alerter-7j5gq\" (UID: \"fbadfc18-65c1-4256-b7da-f153a1a188e9\") " pod="openshift-network-operator/iptables-alerter-7j5gq"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179240    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-sys-fs\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179260    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-run-k8s-cni-cncf-io\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179279    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-run-multus-certs\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179281    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-sysctl-conf\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179301    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-node-log\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179335    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-run-k8s-cni-cncf-io\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179343    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-sys-fs\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179346    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-node-log\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179367    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-log-socket\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179210    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-cni-dir\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179379    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-run-multus-certs\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179395    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4da49179-a694-49ea-a836-7ce5e0fbfb13-kubelet-config\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179415    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-log-socket\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179421    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rk6h\" (UniqueName: \"kubernetes.io/projected/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-kube-api-access-4rk6h\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.179450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179434    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4da49179-a694-49ea-a836-7ce5e0fbfb13-kubelet-config\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179446    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-cnibin\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179460    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-env-overrides\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179469    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6ebba4c-75f7-4b76-a077-c9bbbfd109e0-agent-certs\") pod \"konnectivity-agent-2z92j\" (UID: \"e6ebba4c-75f7-4b76-a077-c9bbbfd109e0\") " pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179494    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-cnibin\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179519    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-var-lib-cni-multus\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179534    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-daemon-config\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179547    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-etc-kubernetes\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179563    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-cnibin\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-cnibin\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179587    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-var-lib-cni-multus\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-systemd-units\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179615    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-etc-kubernetes\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179644    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-systemd-units\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179639    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179650    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-slash\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179703    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvq5v\" (UniqueName: \"kubernetes.io/projected/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-kube-api-access-xvq5v\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179731    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:20.180222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179742    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-slash\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-registration-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179756    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fbadfc18-65c1-4256-b7da-f153a1a188e9-iptables-alerter-script\") pod \"iptables-alerter-7j5gq\" (UID: \"fbadfc18-65c1-4256-b7da-f153a1a188e9\") " pod="openshift-network-operator/iptables-alerter-7j5gq"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179776    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-ovnkube-script-lib\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179780    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-modprobe-d\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.179811    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179855    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-registration-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179876    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-kubernetes\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.179909    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret podName:4da49179-a694-49ea-a836-7ce5e0fbfb13 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:20.679868819 +0000 UTC m=+3.606087240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret") pod "global-pull-secret-syncer-wx589" (UID: "4da49179-a694-49ea-a836-7ce5e0fbfb13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179911    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-modprobe-d\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179945    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-tuned\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179954    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-kubernetes\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179967    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-system-cni-dir\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.179985    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-cni-binary-copy\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180006    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-run-systemd\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180012    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-system-cni-dir\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180037    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-sysctl-d\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.181058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180071    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-run-systemd\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180103    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180124    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-sysctl-d\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebba4c-75f7-4b76-a077-c9bbbfd109e0-konnectivity-ca\") pod \"konnectivity-agent-2z92j\" (UID: \"e6ebba4c-75f7-4b76-a077-c9bbbfd109e0\") " pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180177    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-hostroot\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180226    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-run-ovn-kubernetes\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180260    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180279    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-lib-modules\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180291    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-run-ovn-kubernetes\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180324    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4da49179-a694-49ea-a836-7ce5e0fbfb13-dbus\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180350    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-socket-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180357    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180373    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-systemd\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180375    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-lib-modules\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-kubelet\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180417    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4da49179-a694-49ea-a836-7ce5e0fbfb13-dbus\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:20.181916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180431    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-run-ovn\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180457    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-ovnkube-config\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180466    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-systemd\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180482    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-device-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180492    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-socket-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180506    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-etc-selinux\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180531    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggwtv\" (UniqueName: \"kubernetes.io/projected/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-kube-api-access-ggwtv\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180533    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-cni-binary-copy\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180552    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-socket-dir-parent\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180585    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-device-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180590    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75rh\" (UniqueName: \"kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh\") pod \"network-check-target-mshnf\" (UID: \"f663602d-7308-4423-880f-327ce634f45b\") " pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180589    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-socket-dir-parent\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180620    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-sysconfig\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180644    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-host\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqngb\" (UniqueName: \"kubernetes.io/projected/223bc150-3bae-4e3a-a45e-69865f53a23d-kube-api-access-qqngb\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbadfc18-65c1-4256-b7da-f153a1a188e9-host-slash\") pod \"iptables-alerter-7j5gq\" (UID: \"fbadfc18-65c1-4256-b7da-f153a1a188e9\") " pod="openshift-network-operator/iptables-alerter-7j5gq"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180712    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-sysconfig\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.182481 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180674    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-etc-selinux\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180725    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7km4\" (UniqueName: \"kubernetes.io/projected/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-kube-api-access-j7km4\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180753    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180770    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180788    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5744cdd7-665f-41f2-96b7-150f693100a2-tmp-dir\") pod \"node-resolver-xhmnn\" (UID: \"5744cdd7-665f-41f2-96b7-150f693100a2\") " pod="openshift-dns/node-resolver-xhmnn"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180803    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-conf-dir\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180819    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-etc-openvswitch\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-tmp\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180854    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5744cdd7-665f-41f2-96b7-150f693100a2-hosts-file\") pod \"node-resolver-xhmnn\" (UID: \"5744cdd7-665f-41f2-96b7-150f693100a2\") " pod="openshift-dns/node-resolver-xhmnn"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180869    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-system-cni-dir\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180905    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-system-cni-dir\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180903    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fbadfc18-65c1-4256-b7da-f153a1a188e9-host-slash\") pod \"iptables-alerter-7j5gq\" (UID: \"fbadfc18-65c1-4256-b7da-f153a1a188e9\") " pod="openshift-network-operator/iptables-alerter-7j5gq"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180921    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-var-lib-cni-bin\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180506    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-run-ovn\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180947    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-host\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180953    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-var-lib-kubelet\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180984    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgsl8\" (UniqueName: \"kubernetes.io/projected/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-kube-api-access-lgsl8\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.180536    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-kubelet\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.182994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181037    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-var-lib-cni-bin\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181040    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5744cdd7-665f-41f2-96b7-150f693100a2-hosts-file\") pod \"node-resolver-xhmnn\" (UID: \"5744cdd7-665f-41f2-96b7-150f693100a2\") " pod="openshift-dns/node-resolver-xhmnn"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181058    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-etc-openvswitch\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.181069    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181081    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-host-var-lib-kubelet\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181085    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-var-lib-openvswitch\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181083    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-ovnkube-config\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181110    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-run-openvswitch\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.181128    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs podName:223bc150-3bae-4e3a-a45e-69865f53a23d nodeName:}" failed. No retries permitted until 2026-09-17 19:06:20.681110874 +0000 UTC m=+3.607329276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs") pod "network-metrics-daemon-8gl6r" (UID: "223bc150-3bae-4e3a-a45e-69865f53a23d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181161    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-run-openvswitch\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181288    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6ebba4c-75f7-4b76-a077-c9bbbfd109e0-konnectivity-ca\") pod \"konnectivity-agent-2z92j\" (UID: \"e6ebba4c-75f7-4b76-a077-c9bbbfd109e0\") " pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181403    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5744cdd7-665f-41f2-96b7-150f693100a2-tmp-dir\") pod \"node-resolver-xhmnn\" (UID: \"5744cdd7-665f-41f2-96b7-150f693100a2\") " pod="openshift-dns/node-resolver-xhmnn"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181418    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-multus-conf-dir\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181463    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-ovn-node-metrics-cert\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-cni-netd\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181515    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181539    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-var-lib-kubelet\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.183639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.184129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181600    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-var-lib-openvswitch\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.184129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181678    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-host-cni-netd\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.184129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181708    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.184129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181731    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.184129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181794    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-var-lib-kubelet\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.184129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.181827    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-hostroot\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.184129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.182038    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-cni-binary-copy\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.184129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.182471    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6ebba4c-75f7-4b76-a077-c9bbbfd109e0-agent-certs\") pod \"konnectivity-agent-2z92j\" (UID: \"e6ebba4c-75f7-4b76-a077-c9bbbfd109e0\") " pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:20.184129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.182474    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-etc-tuned\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.184129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.183092    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-tmp\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.184129 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.183581    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-ovn-node-metrics-cert\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.189927 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:20.189900    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3fa8c6c6ae763d57f9ebb8f3758a86e.slice/crio-8267eda367ff8fa0673579184e856ee440e5badf5cf8ad770215ebb62f2a562d WatchSource:0}: Error finding container 8267eda367ff8fa0673579184e856ee440e5badf5cf8ad770215ebb62f2a562d: Status 404 returned error can't find the container with id 8267eda367ff8fa0673579184e856ee440e5badf5cf8ad770215ebb62f2a562d
Sep 17 19:06:20.190318 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:20.190300    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0937b68b17a7b35dfeea96e89aad3eb3.slice/crio-9e5ecc8bf59af8aa867f5e84b8a2d2616914e81ac2e089d433a32ac5bebc36ad WatchSource:0}: Error finding container 9e5ecc8bf59af8aa867f5e84b8a2d2616914e81ac2e089d433a32ac5bebc36ad: Status 404 returned error can't find the container with id 9e5ecc8bf59af8aa867f5e84b8a2d2616914e81ac2e089d433a32ac5bebc36ad
Sep 17 19:06:20.192542 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.192521    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm8xq\" (UniqueName: \"kubernetes.io/projected/fbadfc18-65c1-4256-b7da-f153a1a188e9-kube-api-access-hm8xq\") pod \"iptables-alerter-7j5gq\" (UID: \"fbadfc18-65c1-4256-b7da-f153a1a188e9\") " pod="openshift-network-operator/iptables-alerter-7j5gq"
Sep 17 19:06:20.195291 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.195247    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5nb\" (UniqueName: \"kubernetes.io/projected/5744cdd7-665f-41f2-96b7-150f693100a2-kube-api-access-hp5nb\") pod \"node-resolver-xhmnn\" (UID: \"5744cdd7-665f-41f2-96b7-150f693100a2\") " pod="openshift-dns/node-resolver-xhmnn"
Sep 17 19:06:20.195532 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.195519    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:06:20.195950 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.195928    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rk6h\" (UniqueName: \"kubernetes.io/projected/de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0-kube-api-access-4rk6h\") pod \"aws-ebs-csi-driver-node-gqlrc\" (UID: \"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.201178 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.201131    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggwtv\" (UniqueName: \"kubernetes.io/projected/c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f-kube-api-access-ggwtv\") pod \"multus-additional-cni-plugins-fwmhb\" (UID: \"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f\") " pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.201812 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.201788    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:20.201812 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.201812    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:20.202015 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.201826    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h75rh for pod openshift-network-diagnostics/network-check-target-mshnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:20.202015 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.201911    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh podName:f663602d-7308-4423-880f-327ce634f45b nodeName:}" failed. No retries permitted until 2026-09-17 19:06:20.701892363 +0000 UTC m=+3.628110766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h75rh" (UniqueName: "kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh") pod "network-check-target-mshnf" (UID: "f663602d-7308-4423-880f-327ce634f45b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:20.202896 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.202875    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqngb\" (UniqueName: \"kubernetes.io/projected/223bc150-3bae-4e3a-a45e-69865f53a23d-kube-api-access-qqngb\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:20.203489 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.203462    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgsl8\" (UniqueName: \"kubernetes.io/projected/a31cbf4c-8c80-4d28-b1b4-4bb27aedb301-kube-api-access-lgsl8\") pod \"multus-c5cgz\" (UID: \"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301\") " pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.203731 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.203715    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7km4\" (UniqueName: \"kubernetes.io/projected/e8b40d82-e389-4df4-8ec9-dbf1b57f54a6-kube-api-access-j7km4\") pod \"tuned-gvnfj\" (UID: \"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6\") " pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.204040 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.204023    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvq5v\" (UniqueName: \"kubernetes.io/projected/ea298d9d-0b60-4b56-92e5-2b258c2b5bff-kube-api-access-xvq5v\") pod \"ovnkube-node-cw78b\" (UID: \"ea298d9d-0b60-4b56-92e5-2b258c2b5bff\") " pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.302641 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.302602    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g5mvq"
Sep 17 19:06:20.306821 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.306792    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fwmhb"
Sep 17 19:06:20.313733 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:20.313706    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c8d54f_7756_4625_94b7_806ff2e8db82.slice/crio-8472e81f11657ace158b50461ab84ecb31e3562c3bee29b530e27ba7f7f3dee3 WatchSource:0}: Error finding container 8472e81f11657ace158b50461ab84ecb31e3562c3bee29b530e27ba7f7f3dee3: Status 404 returned error can't find the container with id 8472e81f11657ace158b50461ab84ecb31e3562c3bee29b530e27ba7f7f3dee3
Sep 17 19:06:20.315856 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:20.315838    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a8f197_4b3c_4ddc_8a7e_00d2e0a2e80f.slice/crio-bb768f2dacc39918b640aeec4d99cbe142c55630cd65169cd899da711dc505db WatchSource:0}: Error finding container bb768f2dacc39918b640aeec4d99cbe142c55630cd65169cd899da711dc505db: Status 404 returned error can't find the container with id bb768f2dacc39918b640aeec4d99cbe142c55630cd65169cd899da711dc505db
Sep 17 19:06:20.326132 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.326114    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xhmnn"
Sep 17 19:06:20.335501 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:20.335474    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5744cdd7_665f_41f2_96b7_150f693100a2.slice/crio-a7d2e1b5b09fa57838c80d9dd16f0cdfe7d67783a33df272140adb583e1345f6 WatchSource:0}: Error finding container a7d2e1b5b09fa57838c80d9dd16f0cdfe7d67783a33df272140adb583e1345f6: Status 404 returned error can't find the container with id a7d2e1b5b09fa57838c80d9dd16f0cdfe7d67783a33df272140adb583e1345f6
Sep 17 19:06:20.339739 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.339722    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c5cgz"
Sep 17 19:06:20.347462 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:20.347437    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31cbf4c_8c80_4d28_b1b4_4bb27aedb301.slice/crio-c28348c51877c46d5e025b0019acf5cb7bdc6afcf9fb9b7433fc549cc4585a49 WatchSource:0}: Error finding container c28348c51877c46d5e025b0019acf5cb7bdc6afcf9fb9b7433fc549cc4585a49: Status 404 returned error can't find the container with id c28348c51877c46d5e025b0019acf5cb7bdc6afcf9fb9b7433fc549cc4585a49
Sep 17 19:06:20.355520 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.355501    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7j5gq"
Sep 17 19:06:20.364889 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:20.364862    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbadfc18_65c1_4256_b7da_f153a1a188e9.slice/crio-5e590732c62d3c43ce0a29dc2263f842ffd80694d893c9f7f71a83db6b59e2d9 WatchSource:0}: Error finding container 5e590732c62d3c43ce0a29dc2263f842ffd80694d893c9f7f71a83db6b59e2d9: Status 404 returned error can't find the container with id 5e590732c62d3c43ce0a29dc2263f842ffd80694d893c9f7f71a83db6b59e2d9
Sep 17 19:06:20.372408 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.372389    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:20.379823 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.379804    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:20.380097 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:20.380074    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea298d9d_0b60_4b56_92e5_2b258c2b5bff.slice/crio-049d5c3b9d547eda2f528326f7d3409eabfdf9661436929ccc90d10e769f6c76 WatchSource:0}: Error finding container 049d5c3b9d547eda2f528326f7d3409eabfdf9661436929ccc90d10e769f6c76: Status 404 returned error can't find the container with id 049d5c3b9d547eda2f528326f7d3409eabfdf9661436929ccc90d10e769f6c76
Sep 17 19:06:20.387787 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:20.387763    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ebba4c_75f7_4b76_a077_c9bbbfd109e0.slice/crio-c6c83e55b2eaae09adfbbb76788cc581df88584d03176d15184e77849f34f553 WatchSource:0}: Error finding container c6c83e55b2eaae09adfbbb76788cc581df88584d03176d15184e77849f34f553: Status 404 returned error can't find the container with id c6c83e55b2eaae09adfbbb76788cc581df88584d03176d15184e77849f34f553
Sep 17 19:06:20.402562 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.402541    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc"
Sep 17 19:06:20.407193 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.407173    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gvnfj"
Sep 17 19:06:20.411406 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:20.411386    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8ca7ab_47dc_4c3c_b6bf_4fc64ae34ff0.slice/crio-2aa15222a793d35e06cf96ae072483a6e598b92e35eecbcefae72e3c3b44c7ab WatchSource:0}: Error finding container 2aa15222a793d35e06cf96ae072483a6e598b92e35eecbcefae72e3c3b44c7ab: Status 404 returned error can't find the container with id 2aa15222a793d35e06cf96ae072483a6e598b92e35eecbcefae72e3c3b44c7ab
Sep 17 19:06:20.415414 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:20.415395    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b40d82_e389_4df4_8ec9_dbf1b57f54a6.slice/crio-dff07395392745ad2105a51dab1b60f4245c5daa011591722f79d92548534628 WatchSource:0}: Error finding container dff07395392745ad2105a51dab1b60f4245c5daa011591722f79d92548534628: Status 404 returned error can't find the container with id dff07395392745ad2105a51dab1b60f4245c5daa011591722f79d92548534628
Sep 17 19:06:20.582345 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.582263    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:20.685382 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.685312    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:20.685382 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.685398    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:20.685723 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.685541    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:20.685723 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.685604    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret podName:4da49179-a694-49ea-a836-7ce5e0fbfb13 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:21.685586101 +0000 UTC m=+4.611804518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret") pod "global-pull-secret-syncer-wx589" (UID: "4da49179-a694-49ea-a836-7ce5e0fbfb13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:20.686031 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.686000    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:20.686114 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.686086    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs podName:223bc150-3bae-4e3a-a45e-69865f53a23d nodeName:}" failed. No retries permitted until 2026-09-17 19:06:21.686059992 +0000 UTC m=+4.612278412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs") pod "network-metrics-daemon-8gl6r" (UID: "223bc150-3bae-4e3a-a45e-69865f53a23d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:20.786792 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:20.786754    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75rh\" (UniqueName: \"kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh\") pod \"network-check-target-mshnf\" (UID: \"f663602d-7308-4423-880f-327ce634f45b\") " pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:20.787016 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.787000    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:20.787128 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.787024    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:20.787128 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.787039    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h75rh for pod openshift-network-diagnostics/network-check-target-mshnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:20.787128 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:20.787098    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh podName:f663602d-7308-4423-880f-327ce634f45b nodeName:}" failed. No retries permitted until 2026-09-17 19:06:21.787080425 +0000 UTC m=+4.713298831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h75rh" (UniqueName: "kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh") pod "network-check-target-mshnf" (UID: "f663602d-7308-4423-880f-327ce634f45b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:21.053807 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.053711    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:20 +0000 UTC" deadline="2028-02-23 18:23:20.282049095 +0000 UTC"
Sep 17 19:06:21.053807 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.053754    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12575h16m59.228299329s"
Sep 17 19:06:21.097186 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.097063    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gvnfj" event={"ID":"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6","Type":"ContainerStarted","Data":"dff07395392745ad2105a51dab1b60f4245c5daa011591722f79d92548534628"}
Sep 17 19:06:21.102229 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.102184    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc" event={"ID":"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0","Type":"ContainerStarted","Data":"2aa15222a793d35e06cf96ae072483a6e598b92e35eecbcefae72e3c3b44c7ab"}
Sep 17 19:06:21.113548 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.113511    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7j5gq" event={"ID":"fbadfc18-65c1-4256-b7da-f153a1a188e9","Type":"ContainerStarted","Data":"5e590732c62d3c43ce0a29dc2263f842ffd80694d893c9f7f71a83db6b59e2d9"}
Sep 17 19:06:21.124884 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.124850    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwmhb" event={"ID":"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f","Type":"ContainerStarted","Data":"bb768f2dacc39918b640aeec4d99cbe142c55630cd65169cd899da711dc505db"}
Sep 17 19:06:21.137977 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.137939    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g5mvq" event={"ID":"f8c8d54f-7756-4625-94b7-806ff2e8db82","Type":"ContainerStarted","Data":"8472e81f11657ace158b50461ab84ecb31e3562c3bee29b530e27ba7f7f3dee3"}
Sep 17 19:06:21.145897 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.145859    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2z92j" event={"ID":"e6ebba4c-75f7-4b76-a077-c9bbbfd109e0","Type":"ContainerStarted","Data":"c6c83e55b2eaae09adfbbb76788cc581df88584d03176d15184e77849f34f553"}
Sep 17 19:06:21.159617 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.159584    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b" event={"ID":"ea298d9d-0b60-4b56-92e5-2b258c2b5bff","Type":"ContainerStarted","Data":"049d5c3b9d547eda2f528326f7d3409eabfdf9661436929ccc90d10e769f6c76"}
Sep 17 19:06:21.166500 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.166421    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c5cgz" event={"ID":"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301","Type":"ContainerStarted","Data":"c28348c51877c46d5e025b0019acf5cb7bdc6afcf9fb9b7433fc549cc4585a49"}
Sep 17 19:06:21.171644 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.171579    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xhmnn" event={"ID":"5744cdd7-665f-41f2-96b7-150f693100a2","Type":"ContainerStarted","Data":"a7d2e1b5b09fa57838c80d9dd16f0cdfe7d67783a33df272140adb583e1345f6"}
Sep 17 19:06:21.178117 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.178040    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-225.ec2.internal" event={"ID":"0937b68b17a7b35dfeea96e89aad3eb3","Type":"ContainerStarted","Data":"9e5ecc8bf59af8aa867f5e84b8a2d2616914e81ac2e089d433a32ac5bebc36ad"}
Sep 17 19:06:21.184982 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.184952    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal" event={"ID":"e3fa8c6c6ae763d57f9ebb8f3758a86e","Type":"ContainerStarted","Data":"8267eda367ff8fa0673579184e856ee440e5badf5cf8ad770215ebb62f2a562d"}
Sep 17 19:06:21.383272 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.382518    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:21.568947 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.568919    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:06:21.702069 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.701883    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:21.702069 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.701964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:21.702323 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:21.702098    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:21.702323 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:21.702182    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret podName:4da49179-a694-49ea-a836-7ce5e0fbfb13 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:23.70215554 +0000 UTC m=+6.628373943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret") pod "global-pull-secret-syncer-wx589" (UID: "4da49179-a694-49ea-a836-7ce5e0fbfb13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:21.702903 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:21.702602    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:21.702903 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:21.702656    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs podName:223bc150-3bae-4e3a-a45e-69865f53a23d nodeName:}" failed. No retries permitted until 2026-09-17 19:06:23.702640035 +0000 UTC m=+6.628858441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs") pod "network-metrics-daemon-8gl6r" (UID: "223bc150-3bae-4e3a-a45e-69865f53a23d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:21.802562 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:21.802527    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75rh\" (UniqueName: \"kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh\") pod \"network-check-target-mshnf\" (UID: \"f663602d-7308-4423-880f-327ce634f45b\") " pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:21.802767 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:21.802748    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:21.802845 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:21.802775    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:21.802845 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:21.802790    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h75rh for pod openshift-network-diagnostics/network-check-target-mshnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:21.802958 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:21.802847    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh podName:f663602d-7308-4423-880f-327ce634f45b nodeName:}" failed. No retries permitted until 2026-09-17 19:06:23.802829124 +0000 UTC m=+6.729047544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h75rh" (UniqueName: "kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh") pod "network-check-target-mshnf" (UID: "f663602d-7308-4423-880f-327ce634f45b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:22.054649 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:22.054488    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:01:20 +0000 UTC" deadline="2028-04-23 17:40:36.876429823 +0000 UTC"
Sep 17 19:06:22.054649 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:22.054548    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14014h34m14.82188613s"
Sep 17 19:06:22.078616 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:22.077708    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:22.078616 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:22.077849    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:22.078616 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:22.078307    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:22.078616 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:22.078422    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:22.078616 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:22.078495    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:22.078616 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:22.078563    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:23.717981 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:23.717649    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:23.717981 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:23.717726    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:23.717981 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:23.717871    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:23.717981 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:23.717875    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:23.717981 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:23.717932    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs podName:223bc150-3bae-4e3a-a45e-69865f53a23d nodeName:}" failed. No retries permitted until 2026-09-17 19:06:27.717914402 +0000 UTC m=+10.644132817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs") pod "network-metrics-daemon-8gl6r" (UID: "223bc150-3bae-4e3a-a45e-69865f53a23d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:23.717981 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:23.717952    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret podName:4da49179-a694-49ea-a836-7ce5e0fbfb13 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:27.717941809 +0000 UTC m=+10.644160218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret") pod "global-pull-secret-syncer-wx589" (UID: "4da49179-a694-49ea-a836-7ce5e0fbfb13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:23.818713 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:23.818509    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75rh\" (UniqueName: \"kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh\") pod \"network-check-target-mshnf\" (UID: \"f663602d-7308-4423-880f-327ce634f45b\") " pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:23.818713 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:23.818706    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:23.818963 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:23.818729    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:23.818963 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:23.818742    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h75rh for pod openshift-network-diagnostics/network-check-target-mshnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:23.818963 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:23.818808    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh podName:f663602d-7308-4423-880f-327ce634f45b nodeName:}" failed. No retries permitted until 2026-09-17 19:06:27.818789675 +0000 UTC m=+10.745008099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h75rh" (UniqueName: "kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh") pod "network-check-target-mshnf" (UID: "f663602d-7308-4423-880f-327ce634f45b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:24.078590 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:24.078003    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:24.078590 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:24.078046    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:24.078590 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:24.078174    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:24.078590 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:24.078260    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:24.078590 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:24.078321    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:24.078590 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:24.078255    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:26.077513 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:26.077470    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:26.077513 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:26.077502    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:26.078067 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:26.077486    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:26.078067 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:26.077625    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:26.078067 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:26.077717    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:26.078067 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:26.077786    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:27.753510 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:27.753471    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:27.753949 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:27.753552    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:27.753949 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:27.753691    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:27.753949 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:27.753755    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs podName:223bc150-3bae-4e3a-a45e-69865f53a23d nodeName:}" failed. No retries permitted until 2026-09-17 19:06:35.753734188 +0000 UTC m=+18.679952595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs") pod "network-metrics-daemon-8gl6r" (UID: "223bc150-3bae-4e3a-a45e-69865f53a23d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:27.754191 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:27.754171    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:27.754258 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:27.754232    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret podName:4da49179-a694-49ea-a836-7ce5e0fbfb13 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:35.75421647 +0000 UTC m=+18.680434877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret") pod "global-pull-secret-syncer-wx589" (UID: "4da49179-a694-49ea-a836-7ce5e0fbfb13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:27.855628 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:27.854939    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75rh\" (UniqueName: \"kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh\") pod \"network-check-target-mshnf\" (UID: \"f663602d-7308-4423-880f-327ce634f45b\") " pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:27.855628 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:27.855186    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:27.855628 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:27.855211    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:27.855628 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:27.855224    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h75rh for pod openshift-network-diagnostics/network-check-target-mshnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:27.855628 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:27.855287    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh podName:f663602d-7308-4423-880f-327ce634f45b nodeName:}" failed. No retries permitted until 2026-09-17 19:06:35.855266913 +0000 UTC m=+18.781485316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h75rh" (UniqueName: "kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh") pod "network-check-target-mshnf" (UID: "f663602d-7308-4423-880f-327ce634f45b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:28.077459 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:28.077316    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:28.077459 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:28.077361    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:28.077688 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:28.077460    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:28.078030 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:28.077774    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:28.078030 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:28.077890    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:28.078030 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:28.077968    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:30.077284 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:30.077189    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:30.077284 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:30.077216    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:30.077776 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:30.077189    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:30.077776 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:30.077317    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:30.077776 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:30.077636    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:30.077981 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:30.077950    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:32.077551 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:32.077465    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:32.078065 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:32.077591    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:32.078065 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:32.077597    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:32.078065 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:32.077621    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:32.078065 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:32.077715    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:32.078065 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:32.077768    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:34.077578 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:34.077542    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:34.077578 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:34.077573    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:34.078191 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:34.077576    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:34.078191 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:34.077665    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:34.078191 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:34.077753    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:34.078191 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:34.077865    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:35.814378 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:35.814336    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:35.814889 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:35.814409    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:35.814889 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:35.814504    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:35.814889 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:35.814578    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs podName:223bc150-3bae-4e3a-a45e-69865f53a23d nodeName:}" failed. No retries permitted until 2026-09-17 19:06:51.814563887 +0000 UTC m=+34.740782323 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs") pod "network-metrics-daemon-8gl6r" (UID: "223bc150-3bae-4e3a-a45e-69865f53a23d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:35.814889 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:35.814512    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:35.814889 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:35.814645    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret podName:4da49179-a694-49ea-a836-7ce5e0fbfb13 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:51.814634412 +0000 UTC m=+34.740852839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret") pod "global-pull-secret-syncer-wx589" (UID: "4da49179-a694-49ea-a836-7ce5e0fbfb13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:35.915326 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:35.915272    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75rh\" (UniqueName: \"kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh\") pod \"network-check-target-mshnf\" (UID: \"f663602d-7308-4423-880f-327ce634f45b\") " pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:35.915508 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:35.915455    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:35.915508 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:35.915473    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:35.915508 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:35.915486    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h75rh for pod openshift-network-diagnostics/network-check-target-mshnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:35.915676 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:35.915535    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh podName:f663602d-7308-4423-880f-327ce634f45b nodeName:}" failed. No retries permitted until 2026-09-17 19:06:51.915521785 +0000 UTC m=+34.841740199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h75rh" (UniqueName: "kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh") pod "network-check-target-mshnf" (UID: "f663602d-7308-4423-880f-327ce634f45b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:36.078054 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:36.077973    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:36.078275 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:36.077973    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:36.078275 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:36.078081    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:36.078275 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:36.078196    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:36.078275 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:36.077982    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:36.078454 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:36.078313    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:38.077308 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:38.077269    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:38.077824 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:38.077318    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:38.077824 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:38.077321    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:38.077824 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:38.077405    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:38.077824 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:38.077526    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:38.077824 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:38.077639    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:40.077879 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.077685    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:40.078476 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.077717    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:40.078476 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:40.077963    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:40.078476 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.077728    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:40.078476 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:40.078033    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:40.078476 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:40.078122    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:40.240728 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.240648    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g5mvq" event={"ID":"f8c8d54f-7756-4625-94b7-806ff2e8db82","Type":"ContainerStarted","Data":"4fb83372e7ea0e8df4be61dc475589365b64bc682f59376fdf90773b94b7e1b8"}
Sep 17 19:06:40.242416 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.242237    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2z92j" event={"ID":"e6ebba4c-75f7-4b76-a077-c9bbbfd109e0","Type":"ContainerStarted","Data":"d996086ed7c9b74c6849321e650775f309b3f58e2acf0a3464722eaef6603efa"}
Sep 17 19:06:40.245221 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.245200    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:06:40.245531 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.245511    2567 generic.go:356] "Generic (PLEG): container finished" podID="ea298d9d-0b60-4b56-92e5-2b258c2b5bff" containerID="6f47300fadd324d8227d646f7a9cbb54fc6f9a0a1135b7376661ef050e05c092" exitCode=1
Sep 17 19:06:40.245629 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.245575    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b" event={"ID":"ea298d9d-0b60-4b56-92e5-2b258c2b5bff","Type":"ContainerStarted","Data":"31345bee743542e17344434085570fd779ebddfa6bcf1be940b05e442bdcdf90"}
Sep 17 19:06:40.245678 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.245645    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b" event={"ID":"ea298d9d-0b60-4b56-92e5-2b258c2b5bff","Type":"ContainerStarted","Data":"21ed8ee149fb668a1fafb74d73ab6c4fd268e2d13083ece3534d7fd80f7da54d"}
Sep 17 19:06:40.245714 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.245700    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b" event={"ID":"ea298d9d-0b60-4b56-92e5-2b258c2b5bff","Type":"ContainerStarted","Data":"74442ba017acc012dd188e2a981a64998a530903a064f234e04bc26b68d9a53d"}
Sep 17 19:06:40.245762 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.245749    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b" event={"ID":"ea298d9d-0b60-4b56-92e5-2b258c2b5bff","Type":"ContainerStarted","Data":"b8fd84a660e48dcca853be6dd59969017d765f23b1d4e645ce5fff9769cf9f4c"}
Sep 17 19:06:40.245814 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.245801    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b" event={"ID":"ea298d9d-0b60-4b56-92e5-2b258c2b5bff","Type":"ContainerDied","Data":"6f47300fadd324d8227d646f7a9cbb54fc6f9a0a1135b7376661ef050e05c092"}
Sep 17 19:06:40.245873 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.245838    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b" event={"ID":"ea298d9d-0b60-4b56-92e5-2b258c2b5bff","Type":"ContainerStarted","Data":"c595a708a929c7550369fba93b4032c04ba4584b31924e841e07dec48894886f"}
Sep 17 19:06:40.247076 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.247052    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c5cgz" event={"ID":"a31cbf4c-8c80-4d28-b1b4-4bb27aedb301","Type":"ContainerStarted","Data":"03cb11dc6b384db77ac9b8494dc928180863d2c9430cd9692710310e1d345e72"}
Sep 17 19:06:40.248477 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.248456    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xhmnn" event={"ID":"5744cdd7-665f-41f2-96b7-150f693100a2","Type":"ContainerStarted","Data":"07d82304755e43be924d55d90c9a756ca9075165a669aee4ecae854cf01f65fb"}
Sep 17 19:06:40.249881 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.249862    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-225.ec2.internal" event={"ID":"0937b68b17a7b35dfeea96e89aad3eb3","Type":"ContainerStarted","Data":"83c15e3b0d6e7a199237872a01f11d025e43bbac7a85a5896f41a045025298e6"}
Sep 17 19:06:40.251387 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.251368    2567 generic.go:356] "Generic (PLEG): container finished" podID="e3fa8c6c6ae763d57f9ebb8f3758a86e" containerID="90a59f2aee09ea0dae01821c8c7c2bd162e0437f4c122d8ee074419adcf3b41a" exitCode=0
Sep 17 19:06:40.251463 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.251428    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal" event={"ID":"e3fa8c6c6ae763d57f9ebb8f3758a86e","Type":"ContainerDied","Data":"90a59f2aee09ea0dae01821c8c7c2bd162e0437f4c122d8ee074419adcf3b41a"}
Sep 17 19:06:40.254609 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.253289    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gvnfj" event={"ID":"e8b40d82-e389-4df4-8ec9-dbf1b57f54a6","Type":"ContainerStarted","Data":"ae588afa9811eeea1ff65dd69fc87a71c2f4d3fbd1872579100e874117ee948d"}
Sep 17 19:06:40.255905 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.255884    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc" event={"ID":"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0","Type":"ContainerStarted","Data":"2e5712f0ff09fbdc4b40b78096a9cdcdf1c63b6a36780c352cdec509f9da335b"}
Sep 17 19:06:40.257350 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.257331    2567 generic.go:356] "Generic (PLEG): container finished" podID="c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f" containerID="ffab1c66a075c174cc39ad2c00b91895a985a30cdd9558c5a1bd48ebaf9ba12d" exitCode=0
Sep 17 19:06:40.257429 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.257363    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwmhb" event={"ID":"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f","Type":"ContainerDied","Data":"ffab1c66a075c174cc39ad2c00b91895a985a30cdd9558c5a1bd48ebaf9ba12d"}
Sep 17 19:06:40.260773 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.260735    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g5mvq" podStartSLOduration=2.38112926 podStartE2EDuration="21.260725002s" totalImagesPullingTime="18.879595742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:40.260102011 +0000 UTC m=+23.186320448" watchObservedRunningTime="2026-09-17 19:06:40.260725002 +0000 UTC m=+23.186943425"
Sep 17 19:06:40.307244 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.307186    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xhmnn" podStartSLOduration=2.449399351 podStartE2EDuration="21.307164301s" totalImagesPullingTime="18.85776495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:40.306855206 +0000 UTC m=+23.233073629" watchObservedRunningTime="2026-09-17 19:06:40.307164301 +0000 UTC m=+23.233382728"
Sep 17 19:06:40.331074 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.331030    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2z92j" podStartSLOduration=2.525308008 podStartE2EDuration="21.331014369s" totalImagesPullingTime="18.805706361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:40.331008873 +0000 UTC m=+23.257227306" watchObservedRunningTime="2026-09-17 19:06:40.331014369 +0000 UTC m=+23.257232793"
Sep 17 19:06:40.360496 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.358686    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gvnfj" podStartSLOduration=2.578384925 podStartE2EDuration="21.358663905s" totalImagesPullingTime="18.78027898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:40.358588632 +0000 UTC m=+23.284807055" watchObservedRunningTime="2026-09-17 19:06:40.358663905 +0000 UTC m=+23.284882328"
Sep 17 19:06:40.385973 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.385919    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-225.ec2.internal" podStartSLOduration=21.385898156 podStartE2EDuration="21.385898156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:40.385670661 +0000 UTC m=+23.311889084" watchObservedRunningTime="2026-09-17 19:06:40.385898156 +0000 UTC m=+23.312116580"
Sep 17 19:06:40.423312 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.423258    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c5cgz" podStartSLOduration=2.232296823 podStartE2EDuration="21.423243224s" totalImagesPullingTime="19.190946401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:40.422692565 +0000 UTC m=+23.348910990" watchObservedRunningTime="2026-09-17 19:06:40.423243224 +0000 UTC m=+23.349461647"
Sep 17 19:06:40.809357 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.809332    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 19:06:40.999186 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.999155    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:40.999870 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:40.999847    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:41.061433 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:41.061296    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T19:06:40.809353348Z","UUID":"48d82181-bd06-4884-b80f-65c41071f880","Handler":null,"Name":"","Endpoint":""}
Sep 17 19:06:41.063191 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:41.062990    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 19:06:41.063334 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:41.063203    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 19:06:41.263103 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:41.263068    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal" event={"ID":"e3fa8c6c6ae763d57f9ebb8f3758a86e","Type":"ContainerStarted","Data":"20ffcd7e48f25fdf1e0b7d2116f5c2271558affca55d410eb793d0a6d6ef9cbd"}
Sep 17 19:06:41.265950 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:41.265914    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc" event={"ID":"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0","Type":"ContainerStarted","Data":"8b6d90a10293af02129e0667162b820ba37bff6c19fc8b869f47e36e98bd46fd"}
Sep 17 19:06:41.268028 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:41.267997    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7j5gq" event={"ID":"fbadfc18-65c1-4256-b7da-f153a1a188e9","Type":"ContainerStarted","Data":"a43cca4a38a8adbedf0055af905cb5a22e840297cd2ee662f5bf9c5f0f9a3d71"}
Sep 17 19:06:41.268760 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:41.268740    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:41.269321 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:41.269294    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2z92j"
Sep 17 19:06:41.307991 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:41.307938    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-225.ec2.internal" podStartSLOduration=22.307919591 podStartE2EDuration="22.307919591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:41.282820362 +0000 UTC m=+24.209038787" watchObservedRunningTime="2026-09-17 19:06:41.307919591 +0000 UTC m=+24.234138013"
Sep 17 19:06:41.338079 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:41.337927    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7j5gq" podStartSLOduration=10.951409896 podStartE2EDuration="22.337908784s" totalImagesPullingTime="11.386498888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:41.313990427 +0000 UTC m=+24.240208849" watchObservedRunningTime="2026-09-17 19:06:41.337908784 +0000 UTC m=+24.264127210"
Sep 17 19:06:42.077712 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:42.077627    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:42.077900 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:42.077629    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:42.077900 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:42.077760    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:42.077900 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:42.077629    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:42.077900 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:42.077838    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:42.078084 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:42.077897    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:42.273913 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:42.273886    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:06:42.274346 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:42.274309    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b" event={"ID":"ea298d9d-0b60-4b56-92e5-2b258c2b5bff","Type":"ContainerStarted","Data":"8352abd9512eb11b9b9bf2bfcaf2e98c0bf11a4c0329d2b15a5fe329b1683060"}
Sep 17 19:06:42.276381 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:42.276348    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc" event={"ID":"de8ca7ab-47dc-4c3c-b6bf-4fc64ae34ff0","Type":"ContainerStarted","Data":"b22abb921480dffa289ca7e467528b4533b5da957fb89831ec888549b05a7d6e"}
Sep 17 19:06:42.298560 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:42.298488    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gqlrc" podStartSLOduration=2.897391709 podStartE2EDuration="23.29847031s" totalImagesPullingTime="20.401078601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:42.29798518 +0000 UTC m=+25.224203604" watchObservedRunningTime="2026-09-17 19:06:42.29847031 +0000 UTC m=+25.224688734"
Sep 17 19:06:44.077028 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:44.076992    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:44.077459 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:44.077046    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:44.077459 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:44.076992    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:44.077459 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:44.077123    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:44.077459 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:44.077184    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:44.077459 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:44.077269    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:45.284901 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:45.284675    2567 generic.go:356] "Generic (PLEG): container finished" podID="c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f" containerID="b8de6f3e5f1aea8f80009fbe5f36cc366f5c1adc3e832009f76ac80e7f05977e" exitCode=0
Sep 17 19:06:45.284901 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:45.284732    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwmhb" event={"ID":"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f","Type":"ContainerDied","Data":"b8de6f3e5f1aea8f80009fbe5f36cc366f5c1adc3e832009f76ac80e7f05977e"}
Sep 17 19:06:45.288712 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:45.288696    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:06:45.289025 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:45.289009    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b" event={"ID":"ea298d9d-0b60-4b56-92e5-2b258c2b5bff","Type":"ContainerStarted","Data":"72263f6e3e817b8a77d378643a021e3a42d278c8b73b5ee5380ffbce20c12dc7"}
Sep 17 19:06:45.289352 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:45.289335    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:45.289426 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:45.289412    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:45.289474 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:45.289432    2567 scope.go:117] "RemoveContainer" containerID="6f47300fadd324d8227d646f7a9cbb54fc6f9a0a1135b7376661ef050e05c092"
Sep 17 19:06:45.313791 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:45.313773    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:46.077807 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.077771    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:46.077807 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.077799    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:46.077807 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.077809    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:46.078014 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:46.077895    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:46.078014 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:46.077941    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:46.078075 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:46.078018    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:46.295289 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.295260    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:06:46.295892 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.295694    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b" event={"ID":"ea298d9d-0b60-4b56-92e5-2b258c2b5bff","Type":"ContainerStarted","Data":"2fbe9ba3bf92aed6f37612ce720d195d17a0f409b21dcec778218e240eef7a23"}
Sep 17 19:06:46.296160 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.296082    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:46.298207 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.298183    2567 generic.go:356] "Generic (PLEG): container finished" podID="c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f" containerID="c4ed7febb0457b1a7fa87eb2448fd0fc02ef4d58930fe4795840d57c5593a252" exitCode=0
Sep 17 19:06:46.298286 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.298221    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwmhb" event={"ID":"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f","Type":"ContainerDied","Data":"c4ed7febb0457b1a7fa87eb2448fd0fc02ef4d58930fe4795840d57c5593a252"}
Sep 17 19:06:46.312449 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.312424    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:06:46.357270 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.356943    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b" podStartSLOduration=8.506812146 podStartE2EDuration="27.356919788s" totalImagesPullingTime="18.850107642s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:46.33206037 +0000 UTC m=+29.258278794" watchObservedRunningTime="2026-09-17 19:06:46.356919788 +0000 UTC m=+29.283138213"
Sep 17 19:06:46.602675 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.602643    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wx589"]
Sep 17 19:06:46.602810 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.602793    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:46.602896 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:46.602878    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:46.606438 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.606401    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8gl6r"]
Sep 17 19:06:46.606607 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.606540    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:46.606721 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:46.606653    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:46.607186 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.607163    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mshnf"]
Sep 17 19:06:46.607272 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:46.607249    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:46.607337 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:46.607316    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:47.303323 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:47.303031    2567 generic.go:356] "Generic (PLEG): container finished" podID="c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f" containerID="8d5788b70546c4c050d8e61ff476b036b010698d7e02cecdd425fd0bbf7c09d5" exitCode=0
Sep 17 19:06:47.303323 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:47.303116    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwmhb" event={"ID":"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f","Type":"ContainerDied","Data":"8d5788b70546c4c050d8e61ff476b036b010698d7e02cecdd425fd0bbf7c09d5"}
Sep 17 19:06:48.077019 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:48.076985    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:48.077215 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:48.076986    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:48.077215 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:48.077158    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:48.077324 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:48.077213    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:49.080432 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:49.080362    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:49.080783 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:49.080474    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:50.077356 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:50.077317    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:50.077543 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:50.077431    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:50.077543 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:50.077483    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:50.077618 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:50.077566    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:51.077510 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:51.077470    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:51.077982 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:51.077606    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wx589" podUID="4da49179-a694-49ea-a836-7ce5e0fbfb13"
Sep 17 19:06:51.203638 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:51.203594    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:51.301966 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:51.301925    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:51.835245 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:51.835205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:51.835425 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:51.835278    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:51.835425 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:51.835372    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:51.835425 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:51.835410    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:51.835535 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:51.835438    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret podName:4da49179-a694-49ea-a836-7ce5e0fbfb13 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:23.835418108 +0000 UTC m=+66.761636514 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret") pod "global-pull-secret-syncer-wx589" (UID: "4da49179-a694-49ea-a836-7ce5e0fbfb13") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:51.835535 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:51.835474    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs podName:223bc150-3bae-4e3a-a45e-69865f53a23d nodeName:}" failed. No retries permitted until 2026-09-17 19:07:23.83545786 +0000 UTC m=+66.761676277 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs") pod "network-metrics-daemon-8gl6r" (UID: "223bc150-3bae-4e3a-a45e-69865f53a23d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:51.936227 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:51.936193    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75rh\" (UniqueName: \"kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh\") pod \"network-check-target-mshnf\" (UID: \"f663602d-7308-4423-880f-327ce634f45b\") " pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:51.936410 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:51.936321    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:51.936410 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:51.936334    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:51.936410 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:51.936343    2567 projected.go:196] Error preparing data for projected volume kube-api-access-h75rh for pod openshift-network-diagnostics/network-check-target-mshnf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:51.936410 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:51.936389    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh podName:f663602d-7308-4423-880f-327ce634f45b nodeName:}" failed. No retries permitted until 2026-09-17 19:07:23.93637763 +0000 UTC m=+66.862596034 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h75rh" (UniqueName: "kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh") pod "network-check-target-mshnf" (UID: "f663602d-7308-4423-880f-327ce634f45b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:52.077005 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.076970    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:52.077005 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.076998    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:52.077249 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:52.077088    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mshnf" podUID="f663602d-7308-4423-880f-327ce634f45b"
Sep 17 19:06:52.077249 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:52.077224    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8gl6r" podUID="223bc150-3bae-4e3a-a45e-69865f53a23d"
Sep 17 19:06:52.431156 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.431111    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-225.ec2.internal" event="NodeReady"
Sep 17 19:06:52.431509 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.431250    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 19:06:52.487685 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.487658    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sj5qz"]
Sep 17 19:06:52.503803 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.503773    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wlgn2"]
Sep 17 19:06:52.503939 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.503865    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.508434 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.508406    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 19:06:52.508564 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.508533    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 19:06:52.508721 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.508703    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rj77f\""
Sep 17 19:06:52.520233 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.520211    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wlgn2"]
Sep 17 19:06:52.520337 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.520243    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sj5qz"]
Sep 17 19:06:52.520400 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.520346    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wlgn2"
Sep 17 19:06:52.524859 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.524837    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cmft5\""
Sep 17 19:06:52.525094 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.525079    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 19:06:52.525437 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.525420    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 19:06:52.525589 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.525573    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 19:06:52.641276 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.641236    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp7p7\" (UniqueName: \"kubernetes.io/projected/23034e81-6d79-4221-a617-c31e27bfa62e-kube-api-access-bp7p7\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.641459 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.641290    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.641459 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.641354    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77e6ea87-9a6c-4157-83c6-fc3e356604d2-cert\") pod \"ingress-canary-wlgn2\" (UID: \"77e6ea87-9a6c-4157-83c6-fc3e356604d2\") " pod="openshift-ingress-canary/ingress-canary-wlgn2"
Sep 17 19:06:52.641459 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.641397    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxssr\" (UniqueName: \"kubernetes.io/projected/77e6ea87-9a6c-4157-83c6-fc3e356604d2-kube-api-access-xxssr\") pod \"ingress-canary-wlgn2\" (UID: \"77e6ea87-9a6c-4157-83c6-fc3e356604d2\") " pod="openshift-ingress-canary/ingress-canary-wlgn2"
Sep 17 19:06:52.641459 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.641415    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23034e81-6d79-4221-a617-c31e27bfa62e-tmp-dir\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.641584 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.641510    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/23034e81-6d79-4221-a617-c31e27bfa62e-config-volume\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.742517 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.742486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/23034e81-6d79-4221-a617-c31e27bfa62e-config-volume\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.742630 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.742524    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp7p7\" (UniqueName: \"kubernetes.io/projected/23034e81-6d79-4221-a617-c31e27bfa62e-kube-api-access-bp7p7\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.742630 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.742551    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.742693 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:52.742638    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:52.742738 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:52.742729    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls podName:23034e81-6d79-4221-a617-c31e27bfa62e nodeName:}" failed. No retries permitted until 2026-09-17 19:06:53.242715787 +0000 UTC m=+36.168934188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls") pod "dns-default-sj5qz" (UID: "23034e81-6d79-4221-a617-c31e27bfa62e") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:52.742869 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.742799    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77e6ea87-9a6c-4157-83c6-fc3e356604d2-cert\") pod \"ingress-canary-wlgn2\" (UID: \"77e6ea87-9a6c-4157-83c6-fc3e356604d2\") " pod="openshift-ingress-canary/ingress-canary-wlgn2"
Sep 17 19:06:52.742931 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.742896    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxssr\" (UniqueName: \"kubernetes.io/projected/77e6ea87-9a6c-4157-83c6-fc3e356604d2-kube-api-access-xxssr\") pod \"ingress-canary-wlgn2\" (UID: \"77e6ea87-9a6c-4157-83c6-fc3e356604d2\") " pod="openshift-ingress-canary/ingress-canary-wlgn2"
Sep 17 19:06:52.742931 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.742922    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23034e81-6d79-4221-a617-c31e27bfa62e-tmp-dir\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.743249 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.743229    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23034e81-6d79-4221-a617-c31e27bfa62e-tmp-dir\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.743348 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.743229    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/23034e81-6d79-4221-a617-c31e27bfa62e-config-volume\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.747231 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.747206    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77e6ea87-9a6c-4157-83c6-fc3e356604d2-cert\") pod \"ingress-canary-wlgn2\" (UID: \"77e6ea87-9a6c-4157-83c6-fc3e356604d2\") " pod="openshift-ingress-canary/ingress-canary-wlgn2"
Sep 17 19:06:52.752880 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.752858    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp7p7\" (UniqueName: \"kubernetes.io/projected/23034e81-6d79-4221-a617-c31e27bfa62e-kube-api-access-bp7p7\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:52.753026 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.753009    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxssr\" (UniqueName: \"kubernetes.io/projected/77e6ea87-9a6c-4157-83c6-fc3e356604d2-kube-api-access-xxssr\") pod \"ingress-canary-wlgn2\" (UID: \"77e6ea87-9a6c-4157-83c6-fc3e356604d2\") " pod="openshift-ingress-canary/ingress-canary-wlgn2"
Sep 17 19:06:52.865413 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:52.865384    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wlgn2"
Sep 17 19:06:53.077440 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.077255    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:06:53.080823 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.080803    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:06:53.193432 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.193361    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wlgn2"]
Sep 17 19:06:53.197197 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:53.197174    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e6ea87_9a6c_4157_83c6_fc3e356604d2.slice/crio-82c37517a9563e78cbf7b3ebb3ecbef9c6990fccd0193bd282a748a38fe7779b WatchSource:0}: Error finding container 82c37517a9563e78cbf7b3ebb3ecbef9c6990fccd0193bd282a748a38fe7779b: Status 404 returned error can't find the container with id 82c37517a9563e78cbf7b3ebb3ecbef9c6990fccd0193bd282a748a38fe7779b
Sep 17 19:06:53.213101 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.213080    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"]
Sep 17 19:06:53.241939 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.241915    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-55b68b7566-djg2w"]
Sep 17 19:06:53.242072 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.242040    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:53.244934 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.244911    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 19:06:53.247565 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.247546    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 19:06:53.247710 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.247693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:53.247869 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.247838    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:53.247869 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.247909    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls podName:23034e81-6d79-4221-a617-c31e27bfa62e nodeName:}" failed. No retries permitted until 2026-09-17 19:06:54.247888938 +0000 UTC m=+37.174107359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls") pod "dns-default-sj5qz" (UID: "23034e81-6d79-4221-a617-c31e27bfa62e") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:53.247869 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.247952    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6cvc4\""
Sep 17 19:06:53.247869 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.247969    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 19:06:53.248302 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.248091    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 19:06:53.253265 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.253246    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"]
Sep 17 19:06:53.253349 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.253275    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55b68b7566-djg2w"]
Sep 17 19:06:53.253389 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.253351    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.265246 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.261246    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 19:06:53.265246 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.262188    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 19:06:53.265246 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.262205    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 19:06:53.265246 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.262214    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 19:06:53.265246 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.262222    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 19:06:53.265246 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.262295    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r4dgq\""
Sep 17 19:06:53.265246 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.262459    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 19:06:53.319399 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.319364    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-kj7p7"]
Sep 17 19:06:53.321817 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.321789    2567 generic.go:356] "Generic (PLEG): container finished" podID="c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f" containerID="cd2b288a1b54db109e9341468c137bd09c874734976ff78446b4ec520796c0b7" exitCode=0
Sep 17 19:06:53.331312 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.331291    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwmhb" event={"ID":"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f","Type":"ContainerDied","Data":"cd2b288a1b54db109e9341468c137bd09c874734976ff78446b4ec520796c0b7"}
Sep 17 19:06:53.331398 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.331322    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wlgn2" event={"ID":"77e6ea87-9a6c-4157-83c6-fc3e356604d2","Type":"ContainerStarted","Data":"82c37517a9563e78cbf7b3ebb3ecbef9c6990fccd0193bd282a748a38fe7779b"}
Sep 17 19:06:53.331438 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.331417    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.335610 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.335587    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 19:06:53.335610 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.335602    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 19:06:53.336720 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.336702    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 19:06:53.336720 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.336711    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 19:06:53.336863 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.336721    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-cnnrk\""
Sep 17 19:06:53.342378 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.340583    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-kj7p7"]
Sep 17 19:06:53.344049 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.343836    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 19:06:53.348383 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.348287    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-default-certificate\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.348383 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.348325    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.348383 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.348377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:53.348591 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.348403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-stats-auth\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.348591 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.348425    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg549\" (UniqueName: \"kubernetes.io/projected/5e99e64b-b688-47b4-9271-04fce9af21f3-kube-api-access-dg549\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.348591 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.348474    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c653f4cb-bc89-4470-9d85-f6550ff7f325-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:53.348591 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.348527    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.348591 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.348549    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krmbp\" (UniqueName: \"kubernetes.io/projected/c653f4cb-bc89-4470-9d85-f6550ff7f325-kube-api-access-krmbp\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:53.449095 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.448991    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.449095 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449072    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krmbp\" (UniqueName: \"kubernetes.io/projected/c653f4cb-bc89-4470-9d85-f6550ff7f325-kube-api-access-krmbp\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449150    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/da661868-318a-4f63-8068-f57f60e3f5e7-snapshots\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.449201    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:53.949180477 +0000 UTC m=+36.875398892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : configmap references non-existent config key: service-ca.crt
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449272    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da661868-318a-4f63-8068-f57f60e3f5e7-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449340    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbthg\" (UniqueName: \"kubernetes.io/projected/da661868-318a-4f63-8068-f57f60e3f5e7-kube-api-access-rbthg\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449432    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-default-certificate\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449460    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449507    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da661868-318a-4f63-8068-f57f60e3f5e7-tmp\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.449551    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449556    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449588    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-stats-auth\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449610    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg549\" (UniqueName: \"kubernetes.io/projected/5e99e64b-b688-47b4-9271-04fce9af21f3-kube-api-access-dg549\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449644    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c653f4cb-bc89-4470-9d85-f6550ff7f325-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449672    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da661868-318a-4f63-8068-f57f60e3f5e7-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.449737 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.449692    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da661868-318a-4f63-8068-f57f60e3f5e7-serving-cert\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.450450 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.449807    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:53.450450 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.449853    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls podName:c653f4cb-bc89-4470-9d85-f6550ff7f325 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:53.94983664 +0000 UTC m=+36.876055041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-97zqg" (UID: "c653f4cb-bc89-4470-9d85-f6550ff7f325") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:53.450450 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.449962    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:53.949944287 +0000 UTC m=+36.876162705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : secret "router-metrics-certs-default" not found
Sep 17 19:06:53.450771 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.450750    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c653f4cb-bc89-4470-9d85-f6550ff7f325-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:53.452521 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.452502    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-stats-auth\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.452618 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.452558    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-default-certificate\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.465239 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.465216    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg549\" (UniqueName: \"kubernetes.io/projected/5e99e64b-b688-47b4-9271-04fce9af21f3-kube-api-access-dg549\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.468061 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.468041    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krmbp\" (UniqueName: \"kubernetes.io/projected/c653f4cb-bc89-4470-9d85-f6550ff7f325-kube-api-access-krmbp\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:53.477052 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.477027    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.550322 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.550288    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da661868-318a-4f63-8068-f57f60e3f5e7-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.550322 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.550325    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da661868-318a-4f63-8068-f57f60e3f5e7-serving-cert\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.550521 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.550370    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/da661868-318a-4f63-8068-f57f60e3f5e7-snapshots\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.550521 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.550387    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da661868-318a-4f63-8068-f57f60e3f5e7-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.550521 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.550406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbthg\" (UniqueName: \"kubernetes.io/projected/da661868-318a-4f63-8068-f57f60e3f5e7-kube-api-access-rbthg\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.550661 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.550574    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da661868-318a-4f63-8068-f57f60e3f5e7-tmp\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.550965 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.550935    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da661868-318a-4f63-8068-f57f60e3f5e7-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.551073 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.550999    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da661868-318a-4f63-8068-f57f60e3f5e7-tmp\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.551073 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.551020    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/da661868-318a-4f63-8068-f57f60e3f5e7-snapshots\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.551368 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.551352    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da661868-318a-4f63-8068-f57f60e3f5e7-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.552777 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.552759    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da661868-318a-4f63-8068-f57f60e3f5e7-serving-cert\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.566025 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.566001    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbthg\" (UniqueName: \"kubernetes.io/projected/da661868-318a-4f63-8068-f57f60e3f5e7-kube-api-access-rbthg\") pod \"insights-operator-6f9c644dd9-kj7p7\" (UID: \"da661868-318a-4f63-8068-f57f60e3f5e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.644051 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.644014    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7"
Sep 17 19:06:53.660668 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.660633    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.954404 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.954359    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.954641 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.954435    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:53.954641 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.954488    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:53.954641 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.954558    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:54.954537821 +0000 UTC m=+37.880756244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : configmap references non-existent config key: service-ca.crt
Sep 17 19:06:53.954641 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.954601    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:53.954641 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.954610    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:06:53.954641 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.954642    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls podName:c653f4cb-bc89-4470-9d85-f6550ff7f325 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:54.954631271 +0000 UTC m=+37.880849686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-97zqg" (UID: "c653f4cb-bc89-4470-9d85-f6550ff7f325") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:53.954965 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:53.954679    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:54.954658953 +0000 UTC m=+37.880877371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : secret "router-metrics-certs-default" not found
Sep 17 19:06:53.958484 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:53.958459    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-kj7p7"]
Sep 17 19:06:53.961627 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:53.961603    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda661868_318a_4f63_8068_f57f60e3f5e7.slice/crio-9930870a75c874685024762c24a9dd677dba7074b9d311aaa47c5718e84130de WatchSource:0}: Error finding container 9930870a75c874685024762c24a9dd677dba7074b9d311aaa47c5718e84130de: Status 404 returned error can't find the container with id 9930870a75c874685024762c24a9dd677dba7074b9d311aaa47c5718e84130de
Sep 17 19:06:54.077850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.077774    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:06:54.078013 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.077774    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:06:54.081400 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.081378    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lsqsv\""
Sep 17 19:06:54.081554 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.081529    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-72wlh\""
Sep 17 19:06:54.081871 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.081851    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 19:06:54.082023 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.082003    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:06:54.082567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.082548    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 19:06:54.256756 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.256723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:54.256942 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:54.256930    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:54.257007 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:54.256998    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls podName:23034e81-6d79-4221-a617-c31e27bfa62e nodeName:}" failed. No retries permitted until 2026-09-17 19:06:56.256979243 +0000 UTC m=+39.183197661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls") pod "dns-default-sj5qz" (UID: "23034e81-6d79-4221-a617-c31e27bfa62e") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:54.330897 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.330824    2567 generic.go:356] "Generic (PLEG): container finished" podID="c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f" containerID="779f01ec724aba24027e5aee6fce8b45718fb44cc08f893782b266f1f6ab3484" exitCode=0
Sep 17 19:06:54.331034 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.330908    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwmhb" event={"ID":"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f","Type":"ContainerDied","Data":"779f01ec724aba24027e5aee6fce8b45718fb44cc08f893782b266f1f6ab3484"}
Sep 17 19:06:54.332219 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.332195    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7" event={"ID":"da661868-318a-4f63-8068-f57f60e3f5e7","Type":"ContainerStarted","Data":"9930870a75c874685024762c24a9dd677dba7074b9d311aaa47c5718e84130de"}
Sep 17 19:06:54.964963 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.964739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:54.964963 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:54.964946    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:56.964922858 +0000 UTC m=+39.891141263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : configmap references non-existent config key: service-ca.crt
Sep 17 19:06:54.965832 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.965044    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:54.965832 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:54.965120    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:54.965832 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:54.965224    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:06:54.965832 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:54.965278    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:54.965832 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:54.965290    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:56.965273386 +0000 UTC m=+39.891491803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : secret "router-metrics-certs-default" not found
Sep 17 19:06:54.965832 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:54.965351    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls podName:c653f4cb-bc89-4470-9d85-f6550ff7f325 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:56.965334597 +0000 UTC m=+39.891553004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-97zqg" (UID: "c653f4cb-bc89-4470-9d85-f6550ff7f325") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:55.230876 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:55.230760    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:55.343293 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:55.343251    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fwmhb" event={"ID":"c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f","Type":"ContainerStarted","Data":"e9dd7bba99f5280f24e325b20bbb5c16667ea84bc40ada2a0ab0208f866ec813"}
Sep 17 19:06:55.345483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:55.345441    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:55.371760 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:55.371698    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fwmhb" podStartSLOduration=6.571990104 podStartE2EDuration="36.371678873s" totalImagesPullingTime="29.799688769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:55.36997694 +0000 UTC m=+38.296195389" watchObservedRunningTime="2026-09-17 19:06:55.371678873 +0000 UTC m=+38.297897299"
Sep 17 19:06:55.548827 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:55.548736    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:55.696213 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:55.696169    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:55.805200 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:55.805078    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:56.276458 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:56.276414    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:06:56.276842 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:56.276606    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:06:56.276842 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:56.276689    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls podName:23034e81-6d79-4221-a617-c31e27bfa62e nodeName:}" failed. No retries permitted until 2026-09-17 19:07:00.276666213 +0000 UTC m=+43.202884617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls") pod "dns-default-sj5qz" (UID: "23034e81-6d79-4221-a617-c31e27bfa62e") : secret "dns-default-metrics-tls" not found
Sep 17 19:06:56.982176 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:56.982102    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:06:56.982371 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:56.982230    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:56.982371 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:56.982248    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:56.982371 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:56.982299    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:06:56.982371 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:56.982316    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls podName:c653f4cb-bc89-4470-9d85-f6550ff7f325 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:00.982294254 +0000 UTC m=+43.908512669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-97zqg" (UID: "c653f4cb-bc89-4470-9d85-f6550ff7f325") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:06:56.982537 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:56.982385    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:06:56.982537 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:56.982386    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:00.982372307 +0000 UTC m=+43.908590713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : configmap references non-existent config key: service-ca.crt
Sep 17 19:06:56.982537 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:56.982440    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:00.982426925 +0000 UTC m=+43.908645330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : secret "router-metrics-certs-default" not found
Sep 17 19:06:57.350605 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:57.350511    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7" event={"ID":"da661868-318a-4f63-8068-f57f60e3f5e7","Type":"ContainerStarted","Data":"f89f0ab359dd97b739136951c4d942e95e70094b8192d51d0a60842b9e84339b"}
Sep 17 19:06:57.352189 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:57.352164    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wlgn2" event={"ID":"77e6ea87-9a6c-4157-83c6-fc3e356604d2","Type":"ContainerStarted","Data":"2c000b3c809536c93047882893629296cf8eb870449e86304321ab3114e23718"}
Sep 17 19:06:57.375117 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:57.375069    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7" podStartSLOduration=2.040847346 podStartE2EDuration="4.37505293s" totalImagesPullingTime="2.334205584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:57.374803094 +0000 UTC m=+40.301021518" watchObservedRunningTime="2026-09-17 19:06:57.37505293 +0000 UTC m=+40.301271403"
Sep 17 19:06:57.399347 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:57.399296    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wlgn2" podStartSLOduration=2.306345145 podStartE2EDuration="5.399276959s" totalImagesPullingTime="3.092931814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:57.397978166 +0000 UTC m=+40.324196589" watchObservedRunningTime="2026-09-17 19:06:57.399276959 +0000 UTC m=+40.325495381"
Sep 17 19:06:57.559836 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:57.559801    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:57.659905 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:57.659863    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:58.124172 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.124118    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-2ztxw"]
Sep 17 19:06:58.170824 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.170786    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-2ztxw"]
Sep 17 19:06:58.170979 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.170905    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.177494 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.177457    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:58.183342 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.183324    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 19:06:58.183606 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.183591    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 19:06:58.183675 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.183660    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 19:06:58.184458 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.184442    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q2tm4\""
Sep 17 19:06:58.184795 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.184771    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:06:58.189179 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.189160    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 19:06:58.200997 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.200977    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bljts"]
Sep 17 19:06:58.222814 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.222789    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.227942 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.227925    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vsnnc\""
Sep 17 19:06:58.239998 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.239974    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bljts"]
Sep 17 19:06:58.240109 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.240077    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 19:06:58.248576 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.248552    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 19:06:58.295386 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.295351    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e839db-c4d0-4f68-b9e6-3f8950193307-config\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.295569 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.295391    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f7zj\" (UniqueName: \"kubernetes.io/projected/76e839db-c4d0-4f68-b9e6-3f8950193307-kube-api-access-2f7zj\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.295569 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.295494    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e839db-c4d0-4f68-b9e6-3f8950193307-trusted-ca\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.295569 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.295518    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e839db-c4d0-4f68-b9e6-3f8950193307-serving-cert\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.396425 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.396337    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e839db-c4d0-4f68-b9e6-3f8950193307-trusted-ca\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.396425 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.396400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.396984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.396446    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.396984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.396467    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f7zj\" (UniqueName: \"kubernetes.io/projected/76e839db-c4d0-4f68-b9e6-3f8950193307-kube-api-access-2f7zj\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.396984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.396529    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e839db-c4d0-4f68-b9e6-3f8950193307-serving-cert\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.396984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.396572    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0724dd7f-24c9-4d71-97da-76bb064bd722-crio-socket\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.396984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.396686    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0724dd7f-24c9-4d71-97da-76bb064bd722-data-volume\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.396984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.396718    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl8xb\" (UniqueName: \"kubernetes.io/projected/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-api-access-gl8xb\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.396984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.396747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e839db-c4d0-4f68-b9e6-3f8950193307-config\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.397371 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.397339    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e839db-c4d0-4f68-b9e6-3f8950193307-config\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.397428 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.397395    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e839db-c4d0-4f68-b9e6-3f8950193307-trusted-ca\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.399888 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.399871    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e839db-c4d0-4f68-b9e6-3f8950193307-serving-cert\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.411590 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.411564    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f7zj\" (UniqueName: \"kubernetes.io/projected/76e839db-c4d0-4f68-b9e6-3f8950193307-kube-api-access-2f7zj\") pod \"console-operator-fd4bb46c-2ztxw\" (UID: \"76e839db-c4d0-4f68-b9e6-3f8950193307\") " pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.427712 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.427681    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:58.480656 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.480624    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:06:58.498075 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.498041    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.498273 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:58.498118    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:06:58.498273 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.498133    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.498273 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:58.498211    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm podName:0724dd7f-24c9-4d71-97da-76bb064bd722 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:58.998196019 +0000 UTC m=+41.924414442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bljts" (UID: "0724dd7f-24c9-4d71-97da-76bb064bd722") : configmap "kube-rbac-proxy" not found
Sep 17 19:06:58.498457 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.498299    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0724dd7f-24c9-4d71-97da-76bb064bd722-crio-socket\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.498457 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.498329    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0724dd7f-24c9-4d71-97da-76bb064bd722-data-volume\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.498457 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.498384    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl8xb\" (UniqueName: \"kubernetes.io/projected/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-api-access-gl8xb\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.498457 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:58.498306    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:58.498659 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:58.498472    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls podName:0724dd7f-24c9-4d71-97da-76bb064bd722 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:58.998455141 +0000 UTC m=+41.924673542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bljts" (UID: "0724dd7f-24c9-4d71-97da-76bb064bd722") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:58.498659 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.498518    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0724dd7f-24c9-4d71-97da-76bb064bd722-crio-socket\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.498659 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.498625    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0724dd7f-24c9-4d71-97da-76bb064bd722-data-volume\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.511326 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.511301    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl8xb\" (UniqueName: \"kubernetes.io/projected/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-api-access-gl8xb\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:58.787832 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.787759    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-2ztxw"]
Sep 17 19:06:58.790548 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:06:58.790521    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e839db_c4d0_4f68_b9e6_3f8950193307.slice/crio-b95d5d6660dbe6d1bb75dab4ee53f98f404fc515c3076018dc2bad7c2dbd31dc WatchSource:0}: Error finding container b95d5d6660dbe6d1bb75dab4ee53f98f404fc515c3076018dc2bad7c2dbd31dc: Status 404 returned error can't find the container with id b95d5d6660dbe6d1bb75dab4ee53f98f404fc515c3076018dc2bad7c2dbd31dc
Sep 17 19:06:58.811871 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.811847    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:58.920633 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:58.920596    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:59.002056 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:59.002019    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:59.002237 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:59.002077    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:06:59.002237 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:59.002183    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:06:59.002237 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:59.002225    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:59.002413 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:59.002262    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm podName:0724dd7f-24c9-4d71-97da-76bb064bd722 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:00.002241648 +0000 UTC m=+42.928460066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bljts" (UID: "0724dd7f-24c9-4d71-97da-76bb064bd722") : configmap "kube-rbac-proxy" not found
Sep 17 19:06:59.002413 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:06:59.002281    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls podName:0724dd7f-24c9-4d71-97da-76bb064bd722 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:00.002272894 +0000 UTC m=+42.928491295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bljts" (UID: "0724dd7f-24c9-4d71-97da-76bb064bd722") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:06:59.359111 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:59.359073    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" event={"ID":"76e839db-c4d0-4f68-b9e6-3f8950193307","Type":"ContainerStarted","Data":"b95d5d6660dbe6d1bb75dab4ee53f98f404fc515c3076018dc2bad7c2dbd31dc"}
Sep 17 19:06:59.877178 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:06:59.877132    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xhmnn_5744cdd7-665f-41f2-96b7-150f693100a2/dns-node-resolver/0.log"
Sep 17 19:07:00.011107 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:00.011071    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:07:00.011287 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:00.011208    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:07:00.011287 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:00.011256    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:00.011360 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:00.011287    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:07:00.011360 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:00.011330    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls podName:0724dd7f-24c9-4d71-97da-76bb064bd722 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:02.011313332 +0000 UTC m=+44.937531756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bljts" (UID: "0724dd7f-24c9-4d71-97da-76bb064bd722") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:00.011360 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:00.011346    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm podName:0724dd7f-24c9-4d71-97da-76bb064bd722 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:02.011339387 +0000 UTC m=+44.937557787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bljts" (UID: "0724dd7f-24c9-4d71-97da-76bb064bd722") : configmap "kube-rbac-proxy" not found
Sep 17 19:07:00.312953 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:00.312870    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:07:00.313101 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:00.313042    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:00.313183 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:00.313108    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls podName:23034e81-6d79-4221-a617-c31e27bfa62e nodeName:}" failed. No retries permitted until 2026-09-17 19:07:08.313090663 +0000 UTC m=+51.239309084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls") pod "dns-default-sj5qz" (UID: "23034e81-6d79-4221-a617-c31e27bfa62e") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:00.476559 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:00.476522    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g5mvq_f8c8d54f-7756-4625-94b7-806ff2e8db82/node-ca/0.log"
Sep 17 19:07:00.524701 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:00.524660    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:00.625153 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:00.625113    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:01.019736 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:01.019653    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:01.019736 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:01.019706    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:07:01.020116 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:01.019753    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:01.020116 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:01.019813    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:07:01.020116 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:01.019863    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:07:01.020116 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:01.019882    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:09.019865449 +0000 UTC m=+51.946083850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : secret "router-metrics-certs-default" not found
Sep 17 19:07:01.020116 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:01.019897    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:09.01989037 +0000 UTC m=+51.946108771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : configmap references non-existent config key: service-ca.crt
Sep 17 19:07:01.020116 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:01.019907    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls podName:c653f4cb-bc89-4470-9d85-f6550ff7f325 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:09.019901602 +0000 UTC m=+51.946120002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-97zqg" (UID: "c653f4cb-bc89-4470-9d85-f6550ff7f325") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:07:01.247258 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:01.247229    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:01.292356 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:01.292286    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wlgn2_77e6ea87-9a6c-4157-83c6-fc3e356604d2/serve-healthcheck-canary/0.log"
Sep 17 19:07:01.365578 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:01.365550    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/0.log"
Sep 17 19:07:01.365719 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:01.365589    2567 generic.go:356] "Generic (PLEG): container finished" podID="76e839db-c4d0-4f68-b9e6-3f8950193307" containerID="9d109558fa38264b0cb5eb67e7970e4b94e611abdc2126cf0992c1c7b07ff645" exitCode=255
Sep 17 19:07:01.365719 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:01.365622    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" event={"ID":"76e839db-c4d0-4f68-b9e6-3f8950193307","Type":"ContainerDied","Data":"9d109558fa38264b0cb5eb67e7970e4b94e611abdc2126cf0992c1c7b07ff645"}
Sep 17 19:07:01.365865 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:01.365852    2567 scope.go:117] "RemoveContainer" containerID="9d109558fa38264b0cb5eb67e7970e4b94e611abdc2126cf0992c1c7b07ff645"
Sep 17 19:07:01.402522 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:01.402487    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:01.504396 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:01.504362    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:02.029464 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:02.029422    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:07:02.029854 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:02.029480    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:07:02.029854 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:02.029554    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 19:07:02.029854 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:02.029610    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:02.029854 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:02.029623    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm podName:0724dd7f-24c9-4d71-97da-76bb064bd722 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:06.029606979 +0000 UTC m=+48.955825393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bljts" (UID: "0724dd7f-24c9-4d71-97da-76bb064bd722") : configmap "kube-rbac-proxy" not found
Sep 17 19:07:02.029854 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:02.029651    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls podName:0724dd7f-24c9-4d71-97da-76bb064bd722 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:06.029638109 +0000 UTC m=+48.955856510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bljts" (UID: "0724dd7f-24c9-4d71-97da-76bb064bd722") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:02.369875 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:02.369848    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/1.log"
Sep 17 19:07:02.370334 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:02.370313    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/0.log"
Sep 17 19:07:02.370401 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:02.370355    2567 generic.go:356] "Generic (PLEG): container finished" podID="76e839db-c4d0-4f68-b9e6-3f8950193307" containerID="10aeb2362ac796920b290ed05437847339acb880ee0fa11f2f83035d2111f142" exitCode=255
Sep 17 19:07:02.370448 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:02.370407    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" event={"ID":"76e839db-c4d0-4f68-b9e6-3f8950193307","Type":"ContainerDied","Data":"10aeb2362ac796920b290ed05437847339acb880ee0fa11f2f83035d2111f142"}
Sep 17 19:07:02.370487 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:02.370459    2567 scope.go:117] "RemoveContainer" containerID="9d109558fa38264b0cb5eb67e7970e4b94e611abdc2126cf0992c1c7b07ff645"
Sep 17 19:07:02.370680 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:02.370661    2567 scope.go:117] "RemoveContainer" containerID="10aeb2362ac796920b290ed05437847339acb880ee0fa11f2f83035d2111f142"
Sep 17 19:07:02.370861 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:02.370833    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-2ztxw_openshift-console-operator(76e839db-c4d0-4f68-b9e6-3f8950193307)\"" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" podUID="76e839db-c4d0-4f68-b9e6-3f8950193307"
Sep 17 19:07:03.374498 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:03.374470    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/1.log"
Sep 17 19:07:03.374875 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:03.374806    2567 scope.go:117] "RemoveContainer" containerID="10aeb2362ac796920b290ed05437847339acb880ee0fa11f2f83035d2111f142"
Sep 17 19:07:03.374982 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:03.374961    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-2ztxw_openshift-console-operator(76e839db-c4d0-4f68-b9e6-3f8950193307)\"" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" podUID="76e839db-c4d0-4f68-b9e6-3f8950193307"
Sep 17 19:07:04.843589 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:04.843551    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:04.924275 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:04.924236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:06.061992 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:06.061953    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:07:06.062483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:06.062009    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:07:06.062483 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:06.062165    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:06.062483 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:06.062219    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls podName:0724dd7f-24c9-4d71-97da-76bb064bd722 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:14.062206382 +0000 UTC m=+56.988424783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bljts" (UID: "0724dd7f-24c9-4d71-97da-76bb064bd722") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:06.062673 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:06.062613    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0724dd7f-24c9-4d71-97da-76bb064bd722-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:07:06.147655 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:06.147604    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:08.379455 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:08.379409    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:07:08.379838 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:08.379578    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 19:07:08.379838 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:08.379646    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls podName:23034e81-6d79-4221-a617-c31e27bfa62e nodeName:}" failed. No retries permitted until 2026-09-17 19:07:24.379631045 +0000 UTC m=+67.305849465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls") pod "dns-default-sj5qz" (UID: "23034e81-6d79-4221-a617-c31e27bfa62e") : secret "dns-default-metrics-tls" not found
Sep 17 19:07:08.481161 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:08.481115    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:07:08.481323 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:08.481216    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:07:08.481552 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:08.481540    2567 scope.go:117] "RemoveContainer" containerID="10aeb2362ac796920b290ed05437847339acb880ee0fa11f2f83035d2111f142"
Sep 17 19:07:08.481705 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:08.481687    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-2ztxw_openshift-console-operator(76e839db-c4d0-4f68-b9e6-3f8950193307)\"" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" podUID="76e839db-c4d0-4f68-b9e6-3f8950193307"
Sep 17 19:07:08.973050 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:08.973000    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:09.085072 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.085048    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:09.085248 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.085091    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:07:09.085248 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.085131    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:09.085248 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:09.085202    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 19:07:09.085248 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:09.085211    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 19:07:09.085390 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:09.085258    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:25.08524488 +0000 UTC m=+68.011463297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : secret "router-metrics-certs-default" not found
Sep 17 19:07:09.085390 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:09.085271    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls podName:c653f4cb-bc89-4470-9d85-f6550ff7f325 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:25.085265474 +0000 UTC m=+68.011483874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-97zqg" (UID: "c653f4cb-bc89-4470-9d85-f6550ff7f325") : secret "cluster-monitoring-operator-tls" not found
Sep 17 19:07:09.085390 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:09.085281    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle podName:5e99e64b-b688-47b4-9271-04fce9af21f3 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:25.08527658 +0000 UTC m=+68.011494981 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle") pod "router-default-55b68b7566-djg2w" (UID: "5e99e64b-b688-47b4-9271-04fce9af21f3") : configmap references non-existent config key: service-ca.crt
Sep 17 19:07:09.146719 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.146687    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:09.819257 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.819221    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:09.920160 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.920115    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7"]
Sep 17 19:07:09.955544 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.955508    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7"]
Sep 17 19:07:09.955679 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.955614    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7"
Sep 17 19:07:09.958800 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.958771    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 19:07:09.959836 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.959818    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-94x2q\""
Sep 17 19:07:09.962067 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.962035    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 19:07:09.967121 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.967099    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:09.991632 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:09.991602    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26slj\" (UniqueName: \"kubernetes.io/projected/ccc026e0-deda-423a-bd1d-22e29325633d-kube-api-access-26slj\") pod \"migrator-76794cbd6c-kjzn7\" (UID: \"ccc026e0-deda-423a-bd1d-22e29325633d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7"
Sep 17 19:07:10.067209 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:10.067173    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:10.092080 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:10.091998    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26slj\" (UniqueName: \"kubernetes.io/projected/ccc026e0-deda-423a-bd1d-22e29325633d-kube-api-access-26slj\") pod \"migrator-76794cbd6c-kjzn7\" (UID: \"ccc026e0-deda-423a-bd1d-22e29325633d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7"
Sep 17 19:07:10.104274 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:10.104253    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26slj\" (UniqueName: \"kubernetes.io/projected/ccc026e0-deda-423a-bd1d-22e29325633d-kube-api-access-26slj\") pod \"migrator-76794cbd6c-kjzn7\" (UID: \"ccc026e0-deda-423a-bd1d-22e29325633d\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7"
Sep 17 19:07:10.264450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:10.264340    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7"
Sep 17 19:07:10.265834 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:10.265808    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:10.360590 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:10.360556    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:10.558005 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:10.557969    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7"]
Sep 17 19:07:10.565817 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:10.565785    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc026e0_deda_423a_bd1d_22e29325633d.slice/crio-ff58059ab304797c1bb3b4535459e5a4ace8094575af4fc01e58a1536585cb2c WatchSource:0}: Error finding container ff58059ab304797c1bb3b4535459e5a4ace8094575af4fc01e58a1536585cb2c: Status 404 returned error can't find the container with id ff58059ab304797c1bb3b4535459e5a4ace8094575af4fc01e58a1536585cb2c
Sep 17 19:07:10.609472 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:10.609436    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:10.732314 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:10.732219    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:11.400726 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:11.400694    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7" event={"ID":"ccc026e0-deda-423a-bd1d-22e29325633d","Type":"ContainerStarted","Data":"ff58059ab304797c1bb3b4535459e5a4ace8094575af4fc01e58a1536585cb2c"}
Sep 17 19:07:12.160941 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:12.160906    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:12.259193 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:12.259167    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:12.406718 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:12.406686    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7" event={"ID":"ccc026e0-deda-423a-bd1d-22e29325633d","Type":"ContainerStarted","Data":"6f75d3505a20401989a97c784f9ec77db99eb6b0cea2ec996730fd90d43c1d0c"}
Sep 17 19:07:12.407091 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:12.406747    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7" event={"ID":"ccc026e0-deda-423a-bd1d-22e29325633d","Type":"ContainerStarted","Data":"e5d686a83bdeeaa9f1d6e8214ff0ffe6613ac4c56f6ccfdc7481d567228d635b"}
Sep 17 19:07:12.426491 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:12.426404    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-kjzn7" podStartSLOduration=2.367941997 podStartE2EDuration="3.426389987s" totalImagesPullingTime="1.05844799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:12.425247116 +0000 UTC m=+55.351465539" watchObservedRunningTime="2026-09-17 19:07:12.426389987 +0000 UTC m=+55.352608410"
Sep 17 19:07:12.566664 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:12.566634    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:13.298866 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:13.298819    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:14.127499 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:14.127468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:07:14.127890 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:14.127603    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:14.127890 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:14.127665    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls podName:0724dd7f-24c9-4d71-97da-76bb064bd722 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:30.12764851 +0000 UTC m=+73.053866918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bljts" (UID: "0724dd7f-24c9-4d71-97da-76bb064bd722") : secret "insights-runtime-extractor-tls" not found
Sep 17 19:07:14.252079 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:14.252028    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:14.352902 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:14.352860    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:14.457261 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:14.457160    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:14.713993 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:14.713922    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:14.816578 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:14.816549    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:14.934245 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:14.934205    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:15.037254 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:15.037172    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:15.500343 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:15.500308    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:15.601299 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:15.601267    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:15.991301 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:15.991264    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:16.158301 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:16.158263    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:16.361153 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:16.361109    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:16.564093 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:16.564060    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:16.752150 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:16.752058    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:16.957281 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:16.957251    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:17.942245 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:17.942204    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:18.014329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:18.014300    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:18.154061 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:18.154029    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:18.317307 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:18.317236    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cw78b"
Sep 17 19:07:18.409634 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:18.409604    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:18.626044 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:18.626005    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:18.789515 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:18.789481    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:18.967032 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:18.966955    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:19.967334 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:19.967300    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:20.194152 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:20.194089    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:20.575720 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:20.575682    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:20.680932 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:20.680903    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:20.799155 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:20.799096    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:20.981327 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:20.981298    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:21.963946 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:21.963919    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:22.408440 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:22.408402    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:22.553225 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:22.553183    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:22.761674 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:22.761584    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:23.081317 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.081247    2567 scope.go:117] "RemoveContainer" containerID="10aeb2362ac796920b290ed05437847339acb880ee0fa11f2f83035d2111f142"
Sep 17 19:07:23.165958 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.165907    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:23.447555 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.447524    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:07:23.447998 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.447951    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/1.log"
Sep 17 19:07:23.447998 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.447979    2567 generic.go:356] "Generic (PLEG): container finished" podID="76e839db-c4d0-4f68-b9e6-3f8950193307" containerID="55882f2db0912a86a74014202c396c46af6a359c551b8db3c92ef5b270f29e55" exitCode=255
Sep 17 19:07:23.448086 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.448007    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" event={"ID":"76e839db-c4d0-4f68-b9e6-3f8950193307","Type":"ContainerDied","Data":"55882f2db0912a86a74014202c396c46af6a359c551b8db3c92ef5b270f29e55"}
Sep 17 19:07:23.448086 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.448069    2567 scope.go:117] "RemoveContainer" containerID="10aeb2362ac796920b290ed05437847339acb880ee0fa11f2f83035d2111f142"
Sep 17 19:07:23.448479 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.448459    2567 scope.go:117] "RemoveContainer" containerID="55882f2db0912a86a74014202c396c46af6a359c551b8db3c92ef5b270f29e55"
Sep 17 19:07:23.448628 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:23.448605    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-2ztxw_openshift-console-operator(76e839db-c4d0-4f68-b9e6-3f8950193307)\"" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" podUID="76e839db-c4d0-4f68-b9e6-3f8950193307"
Sep 17 19:07:23.761331 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.761229    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:23.910450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.910410    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:07:23.910581 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.910505    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:07:23.915512 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.915489    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:07:23.916428 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.916411    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:07:23.923469 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.923427    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4da49179-a694-49ea-a836-7ce5e0fbfb13-original-pull-secret\") pod \"global-pull-secret-syncer-wx589\" (UID: \"4da49179-a694-49ea-a836-7ce5e0fbfb13\") " pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:07:23.923578 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.923470    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/223bc150-3bae-4e3a-a45e-69865f53a23d-metrics-certs\") pod \"network-metrics-daemon-8gl6r\" (UID: \"223bc150-3bae-4e3a-a45e-69865f53a23d\") " pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:07:23.986847 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:23.986807    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wx589"
Sep 17 19:07:24.011224 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.011194    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h75rh\" (UniqueName: \"kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh\") pod \"network-check-target-mshnf\" (UID: \"f663602d-7308-4423-880f-327ce634f45b\") " pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:07:24.014310 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.014257    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 19:07:24.025011 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.024988    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 19:07:24.034715 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.034683    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h75rh\" (UniqueName: \"kubernetes.io/projected/f663602d-7308-4423-880f-327ce634f45b-kube-api-access-h75rh\") pod \"network-check-target-mshnf\" (UID: \"f663602d-7308-4423-880f-327ce634f45b\") " pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:07:24.096749 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.096692    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lsqsv\""
Sep 17 19:07:24.099721 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.099701    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8gl6r"
Sep 17 19:07:24.101572 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.101555    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-72wlh\""
Sep 17 19:07:24.106062 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.106045    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:07:24.295201 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.295109    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wx589"]
Sep 17 19:07:24.298965 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:24.298923    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da49179_a694_49ea_a836_7ce5e0fbfb13.slice/crio-77bbdfbca5a157e6a6c8395d9595fa7ea176bc4f1658bd7f13d1c91172f6dae7 WatchSource:0}: Error finding container 77bbdfbca5a157e6a6c8395d9595fa7ea176bc4f1658bd7f13d1c91172f6dae7: Status 404 returned error can't find the container with id 77bbdfbca5a157e6a6c8395d9595fa7ea176bc4f1658bd7f13d1c91172f6dae7
Sep 17 19:07:24.416628 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.416598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:07:24.418984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.418947    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/23034e81-6d79-4221-a617-c31e27bfa62e-metrics-tls\") pod \"dns-default-sj5qz\" (UID: \"23034e81-6d79-4221-a617-c31e27bfa62e\") " pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:07:24.420070 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.420047    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8gl6r"]
Sep 17 19:07:24.423505 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:24.423484    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223bc150_3bae_4e3a_a45e_69865f53a23d.slice/crio-f3f57c9a701c1eba2aff70c8939536722d43f867dc72d08793c65a2e0bbd4c3d WatchSource:0}: Error finding container f3f57c9a701c1eba2aff70c8939536722d43f867dc72d08793c65a2e0bbd4c3d: Status 404 returned error can't find the container with id f3f57c9a701c1eba2aff70c8939536722d43f867dc72d08793c65a2e0bbd4c3d
Sep 17 19:07:24.438068 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.438036    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:24.441662 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.441641    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mshnf"]
Sep 17 19:07:24.444592 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:24.444562    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf663602d_7308_4423_880f_327ce634f45b.slice/crio-be0e929c58b13684c9caff88e172b92abb2968f14bcc393e69a6b5cf7d9649a7 WatchSource:0}: Error finding container be0e929c58b13684c9caff88e172b92abb2968f14bcc393e69a6b5cf7d9649a7: Status 404 returned error can't find the container with id be0e929c58b13684c9caff88e172b92abb2968f14bcc393e69a6b5cf7d9649a7
Sep 17 19:07:24.452191 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.452169    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mshnf" event={"ID":"f663602d-7308-4423-880f-327ce634f45b","Type":"ContainerStarted","Data":"be0e929c58b13684c9caff88e172b92abb2968f14bcc393e69a6b5cf7d9649a7"}
Sep 17 19:07:24.453346 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.453324    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8gl6r" event={"ID":"223bc150-3bae-4e3a-a45e-69865f53a23d","Type":"ContainerStarted","Data":"f3f57c9a701c1eba2aff70c8939536722d43f867dc72d08793c65a2e0bbd4c3d"}
Sep 17 19:07:24.454759 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.454742    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:07:24.455836 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.455817    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wx589" event={"ID":"4da49179-a694-49ea-a836-7ce5e0fbfb13","Type":"ContainerStarted","Data":"77bbdfbca5a157e6a6c8395d9595fa7ea176bc4f1658bd7f13d1c91172f6dae7"}
Sep 17 19:07:24.589606 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.589499    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:24.658487 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.658458    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rj77f\""
Sep 17 19:07:24.665836 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.665814    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:07:24.787603 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.787568    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:24.961425 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.961345    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sj5qz"]
Sep 17 19:07:24.965790 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:24.965755    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23034e81_6d79_4221_a617_c31e27bfa62e.slice/crio-5e414a9edbbc8cd3b30ac39e4cb04a5335c62a4628b111d35dd748ab38dec0f7 WatchSource:0}: Error finding container 5e414a9edbbc8cd3b30ac39e4cb04a5335c62a4628b111d35dd748ab38dec0f7: Status 404 returned error can't find the container with id 5e414a9edbbc8cd3b30ac39e4cb04a5335c62a4628b111d35dd748ab38dec0f7
Sep 17 19:07:24.974705 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:24.974674    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:25.122754 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.122710    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:25.122950 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.122773    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:07:25.122950 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.122829    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:25.123658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.123634    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e99e64b-b688-47b4-9271-04fce9af21f3-service-ca-bundle\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:25.125598 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.125572    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c653f4cb-bc89-4470-9d85-f6550ff7f325-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-97zqg\" (UID: \"c653f4cb-bc89-4470-9d85-f6550ff7f325\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:07:25.126212 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.126170    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e99e64b-b688-47b4-9271-04fce9af21f3-metrics-certs\") pod \"router-default-55b68b7566-djg2w\" (UID: \"5e99e64b-b688-47b4-9271-04fce9af21f3\") " pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:25.354184 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.354084    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6cvc4\""
Sep 17 19:07:25.361856 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.361827    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"
Sep 17 19:07:25.363502 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.363482    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r4dgq\""
Sep 17 19:07:25.370972 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.370951    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:25.411499 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.411360    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:25.462558 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:25.462520    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sj5qz" event={"ID":"23034e81-6d79-4221-a617-c31e27bfa62e","Type":"ContainerStarted","Data":"5e414a9edbbc8cd3b30ac39e4cb04a5335c62a4628b111d35dd748ab38dec0f7"}
Sep 17 19:07:26.001340 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:26.001177    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-55b68b7566-djg2w"]
Sep 17 19:07:26.009852 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:26.009817    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc653f4cb_bc89_4470_9d85_f6550ff7f325.slice/crio-85b7f52e2c7fece55800c40a2c5533d900ddff588e42522eb049fdc13004dfbb WatchSource:0}: Error finding container 85b7f52e2c7fece55800c40a2c5533d900ddff588e42522eb049fdc13004dfbb: Status 404 returned error can't find the container with id 85b7f52e2c7fece55800c40a2c5533d900ddff588e42522eb049fdc13004dfbb
Sep 17 19:07:26.011815 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:26.011787    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e99e64b_b688_47b4_9271_04fce9af21f3.slice/crio-ea9666bef8347fef65e8337f6926c39ece20f7ecd8bb4e6c4ae84cb3a1b7948b WatchSource:0}: Error finding container ea9666bef8347fef65e8337f6926c39ece20f7ecd8bb4e6c4ae84cb3a1b7948b: Status 404 returned error can't find the container with id ea9666bef8347fef65e8337f6926c39ece20f7ecd8bb4e6c4ae84cb3a1b7948b
Sep 17 19:07:26.014300 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:26.014270    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg"]
Sep 17 19:07:26.470233 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:26.470171    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8gl6r" event={"ID":"223bc150-3bae-4e3a-a45e-69865f53a23d","Type":"ContainerStarted","Data":"f357816e66f2ed43ead7765629b09ae26e12bb9132f16c3d1fefff7399cacd70"}
Sep 17 19:07:26.470233 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:26.470236    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8gl6r" event={"ID":"223bc150-3bae-4e3a-a45e-69865f53a23d","Type":"ContainerStarted","Data":"8fbb072c5a77e83694a56faa93e06ef4827b62c6d16aac46514bd77fedd75205"}
Sep 17 19:07:26.472555 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:26.472527    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg" event={"ID":"c653f4cb-bc89-4470-9d85-f6550ff7f325","Type":"ContainerStarted","Data":"85b7f52e2c7fece55800c40a2c5533d900ddff588e42522eb049fdc13004dfbb"}
Sep 17 19:07:26.476611 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:26.476577    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55b68b7566-djg2w" event={"ID":"5e99e64b-b688-47b4-9271-04fce9af21f3","Type":"ContainerStarted","Data":"9434c79b2c36f4d1d3e682e5bc496f03f15876abf8f1709f3f1eab8d46c3b2cf"}
Sep 17 19:07:26.476764 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:26.476634    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-55b68b7566-djg2w" event={"ID":"5e99e64b-b688-47b4-9271-04fce9af21f3","Type":"ContainerStarted","Data":"ea9666bef8347fef65e8337f6926c39ece20f7ecd8bb4e6c4ae84cb3a1b7948b"}
Sep 17 19:07:26.516076 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:26.515956    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8gl6r" podStartSLOduration=65.983259144 podStartE2EDuration="1m7.515937516s" totalImagesPullingTime="1.532678372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:26.494834273 +0000 UTC m=+69.421052697" watchObservedRunningTime="2026-09-17 19:07:26.515937516 +0000 UTC m=+69.442155938"
Sep 17 19:07:27.371804 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:27.371763    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:27.374773 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:27.374748    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:27.413912 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:27.413826    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-55b68b7566-djg2w" podStartSLOduration=34.413801278 podStartE2EDuration="34.413801278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:26.516807421 +0000 UTC m=+69.443025844" watchObservedRunningTime="2026-09-17 19:07:27.413801278 +0000 UTC m=+70.340019692"
Sep 17 19:07:27.481343 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:27.481303    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:27.482670 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:27.482643    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-55b68b7566-djg2w"
Sep 17 19:07:28.481848 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:28.481801    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:07:28.482378 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:28.481872    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:07:28.482378 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:28.482312    2567 scope.go:117] "RemoveContainer" containerID="55882f2db0912a86a74014202c396c46af6a359c551b8db3c92ef5b270f29e55"
Sep 17 19:07:28.482507 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:28.482482    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-2ztxw_openshift-console-operator(76e839db-c4d0-4f68-b9e6-3f8950193307)\"" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" podUID="76e839db-c4d0-4f68-b9e6-3f8950193307"
Sep 17 19:07:30.172240 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.172038    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:07:30.175349 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.175303    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0724dd7f-24c9-4d71-97da-76bb064bd722-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bljts\" (UID: \"0724dd7f-24c9-4d71-97da-76bb064bd722\") " pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:07:30.306892 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.306811    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:30.334352 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.334321    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vsnnc\""
Sep 17 19:07:30.342633 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.342602    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bljts"
Sep 17 19:07:30.407640 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.407559    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:30.493958 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.493922    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mshnf" event={"ID":"f663602d-7308-4423-880f-327ce634f45b","Type":"ContainerStarted","Data":"875627efdd9155e3fd2b53b01eb9c484b57af82f605475f689a8b86965f81b66"}
Sep 17 19:07:30.494180 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.494020    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:07:30.495858 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.495834    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sj5qz" event={"ID":"23034e81-6d79-4221-a617-c31e27bfa62e","Type":"ContainerStarted","Data":"6f7883cf5e424906d56a7991058f122570c016869ec12b5cfd29bde3f78e6443"}
Sep 17 19:07:30.495985 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.495873    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sj5qz" event={"ID":"23034e81-6d79-4221-a617-c31e27bfa62e","Type":"ContainerStarted","Data":"a6601972d34db2886077c0b4d2cdc403f68975fe662ea77ba5fa2dd00cd81aeb"}
Sep 17 19:07:30.495985 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.495903    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:07:30.497551 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.497517    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg" event={"ID":"c653f4cb-bc89-4470-9d85-f6550ff7f325","Type":"ContainerStarted","Data":"d5e079abf25cf0a1a6b806b0c4bd42aadae821670e04d5e2a35501de1f5d1abc"}
Sep 17 19:07:30.499030 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.499008    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wx589" event={"ID":"4da49179-a694-49ea-a836-7ce5e0fbfb13","Type":"ContainerStarted","Data":"8cee511959d85ebbeea54db8b1e948fe650c663892c42294f0a954388f4a740c"}
Sep 17 19:07:30.527046 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.526972    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mshnf" podStartSLOduration=65.831076452 podStartE2EDuration="1m11.52695237s" totalImagesPullingTime="5.695875918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:30.519436378 +0000 UTC m=+73.445654803" watchObservedRunningTime="2026-09-17 19:07:30.52695237 +0000 UTC m=+73.453170792"
Sep 17 19:07:30.582470 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.582437    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:30.587173 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.587104    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wx589" podStartSLOduration=65.742024388 podStartE2EDuration="1m11.587087075s" totalImagesPullingTime="5.845062687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:30.55196237 +0000 UTC m=+73.478180793" watchObservedRunningTime="2026-09-17 19:07:30.587087075 +0000 UTC m=+73.513305497"
Sep 17 19:07:30.587550 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.587527    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sj5qz" podStartSLOduration=33.423485011 podStartE2EDuration="38.5875195s" totalImagesPullingTime="5.164034489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:30.586155531 +0000 UTC m=+73.512373944" watchObservedRunningTime="2026-09-17 19:07:30.5875195 +0000 UTC m=+73.513737923"
Sep 17 19:07:30.649020 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.648952    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-97zqg" podStartSLOduration=33.526495546 podStartE2EDuration="37.648922061s" totalImagesPullingTime="4.122426515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:30.616507301 +0000 UTC m=+73.542725735" watchObservedRunningTime="2026-09-17 19:07:30.648922061 +0000 UTC m=+73.575140485"
Sep 17 19:07:30.649756 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.649734    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bljts"]
Sep 17 19:07:30.653178 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:30.653122    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0724dd7f_24c9_4d71_97da_76bb064bd722.slice/crio-4195631e6b6f00068dcefbc4ba46a5abd23b8c335c984424151a3e484dabbb7a WatchSource:0}: Error finding container 4195631e6b6f00068dcefbc4ba46a5abd23b8c335c984424151a3e484dabbb7a: Status 404 returned error can't find the container with id 4195631e6b6f00068dcefbc4ba46a5abd23b8c335c984424151a3e484dabbb7a
Sep 17 19:07:30.690081 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.690045    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:30.809074 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.809036    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:30.984581 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:30.984498    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:31.503934 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:31.503895    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bljts" event={"ID":"0724dd7f-24c9-4d71-97da-76bb064bd722","Type":"ContainerStarted","Data":"0a4711c89979619343f43464f520baec629bf45341876ce60b6cdd58c800f191"}
Sep 17 19:07:31.504438 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:31.503959    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bljts" event={"ID":"0724dd7f-24c9-4d71-97da-76bb064bd722","Type":"ContainerStarted","Data":"82be28d81e662043b431576edac5d3522a0a356feac17515bb800ee67427980c"}
Sep 17 19:07:31.504438 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:31.503984    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bljts" event={"ID":"0724dd7f-24c9-4d71-97da-76bb064bd722","Type":"ContainerStarted","Data":"4195631e6b6f00068dcefbc4ba46a5abd23b8c335c984424151a3e484dabbb7a"}
Sep 17 19:07:33.444490 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.444457    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw"]
Sep 17 19:07:33.450585 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.450556    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw"
Sep 17 19:07:33.453859 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.453828    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 19:07:33.454815 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.454795    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-fkgf9\""
Sep 17 19:07:33.455019 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.455003    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 19:07:33.455084 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.455030    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 19:07:33.455178 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.455158    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 19:07:33.460542 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.460521    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw"]
Sep 17 19:07:33.514108 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.514073    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bljts" event={"ID":"0724dd7f-24c9-4d71-97da-76bb064bd722","Type":"ContainerStarted","Data":"27ac049a3b918295104f82077a785966434ff620d50fb2c43243ad991520e74e"}
Sep 17 19:07:33.548744 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.548686    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bljts" podStartSLOduration=33.742632781 podStartE2EDuration="35.548668243s" totalImagesPullingTime="1.806035462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:33.546064676 +0000 UTC m=+76.472283089" watchObservedRunningTime="2026-09-17 19:07:33.548668243 +0000 UTC m=+76.474886668"
Sep 17 19:07:33.585766 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.585734    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-557d646d44-6wm5d"]
Sep 17 19:07:33.587430 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.587401    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:33.590256 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.590236    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.598343 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.598315    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb638639-51b4-47b4-bcd2-43ce0e5872c4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59df778558-bnxjw\" (UID: \"bb638639-51b4-47b4-bcd2-43ce0e5872c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw"
Sep 17 19:07:33.598483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.598361    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqk8d\" (UniqueName: \"kubernetes.io/projected/bb638639-51b4-47b4-bcd2-43ce0e5872c4-kube-api-access-xqk8d\") pod \"managed-serviceaccount-addon-agent-59df778558-bnxjw\" (UID: \"bb638639-51b4-47b4-bcd2-43ce0e5872c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw"
Sep 17 19:07:33.600337 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.600313    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4tjxz\""
Sep 17 19:07:33.600435 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.600319    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 19:07:33.600487 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.600459    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 19:07:33.600627 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.600615    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 19:07:33.603219 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.603195    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-557d646d44-6wm5d"]
Sep 17 19:07:33.609503 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.609477    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 19:07:33.677382 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.677339    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:33.699376 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.699288    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c235293a-4120-48f2-8a31-21b5209b67a0-bound-sa-token\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.699376 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.699327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgq6p\" (UniqueName: \"kubernetes.io/projected/c235293a-4120-48f2-8a31-21b5209b67a0-kube-api-access-lgq6p\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.699376 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.699354    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c235293a-4120-48f2-8a31-21b5209b67a0-ca-trust-extracted\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.699609 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.699410    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c235293a-4120-48f2-8a31-21b5209b67a0-registry-certificates\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.699609 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.699428    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c235293a-4120-48f2-8a31-21b5209b67a0-installation-pull-secrets\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.699609 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.699450    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c235293a-4120-48f2-8a31-21b5209b67a0-image-registry-private-configuration\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.699609 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.699486    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c235293a-4120-48f2-8a31-21b5209b67a0-trusted-ca\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.699609 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.699555    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb638639-51b4-47b4-bcd2-43ce0e5872c4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59df778558-bnxjw\" (UID: \"bb638639-51b4-47b4-bcd2-43ce0e5872c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw"
Sep 17 19:07:33.699609 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.699587    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqk8d\" (UniqueName: \"kubernetes.io/projected/bb638639-51b4-47b4-bcd2-43ce0e5872c4-kube-api-access-xqk8d\") pod \"managed-serviceaccount-addon-agent-59df778558-bnxjw\" (UID: \"bb638639-51b4-47b4-bcd2-43ce0e5872c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw"
Sep 17 19:07:33.699609 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.699601    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c235293a-4120-48f2-8a31-21b5209b67a0-registry-tls\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.702171 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.702132    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bb638639-51b4-47b4-bcd2-43ce0e5872c4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59df778558-bnxjw\" (UID: \"bb638639-51b4-47b4-bcd2-43ce0e5872c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw"
Sep 17 19:07:33.717752 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.717712    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqk8d\" (UniqueName: \"kubernetes.io/projected/bb638639-51b4-47b4-bcd2-43ce0e5872c4-kube-api-access-xqk8d\") pod \"managed-serviceaccount-addon-agent-59df778558-bnxjw\" (UID: \"bb638639-51b4-47b4-bcd2-43ce0e5872c4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw"
Sep 17 19:07:33.765987 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.765944    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw"
Sep 17 19:07:33.800052 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.800016    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c235293a-4120-48f2-8a31-21b5209b67a0-registry-tls\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.800269 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.800062    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c235293a-4120-48f2-8a31-21b5209b67a0-bound-sa-token\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.800269 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.800086    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgq6p\" (UniqueName: \"kubernetes.io/projected/c235293a-4120-48f2-8a31-21b5209b67a0-kube-api-access-lgq6p\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.800269 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.800123    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c235293a-4120-48f2-8a31-21b5209b67a0-ca-trust-extracted\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.800269 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.800165    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c235293a-4120-48f2-8a31-21b5209b67a0-registry-certificates\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.800269 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.800186    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c235293a-4120-48f2-8a31-21b5209b67a0-installation-pull-secrets\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.800269 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.800216    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c235293a-4120-48f2-8a31-21b5209b67a0-image-registry-private-configuration\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.800269 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.800240    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c235293a-4120-48f2-8a31-21b5209b67a0-trusted-ca\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.800631 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.800566    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c235293a-4120-48f2-8a31-21b5209b67a0-ca-trust-extracted\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.801313 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.801260    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c235293a-4120-48f2-8a31-21b5209b67a0-registry-certificates\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.801477 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.801403    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c235293a-4120-48f2-8a31-21b5209b67a0-trusted-ca\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.802845 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.802821    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c235293a-4120-48f2-8a31-21b5209b67a0-image-registry-private-configuration\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.802997 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.802897    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c235293a-4120-48f2-8a31-21b5209b67a0-registry-tls\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.803063 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.802997    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c235293a-4120-48f2-8a31-21b5209b67a0-installation-pull-secrets\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.812763 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.812722    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c235293a-4120-48f2-8a31-21b5209b67a0-bound-sa-token\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.813166 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.813128    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgq6p\" (UniqueName: \"kubernetes.io/projected/c235293a-4120-48f2-8a31-21b5209b67a0-kube-api-access-lgq6p\") pod \"image-registry-557d646d44-6wm5d\" (UID: \"c235293a-4120-48f2-8a31-21b5209b67a0\") " pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.814767 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.814727    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:33.899646 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.899613    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:33.948084 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:33.948040    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:34.175328 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:34.175272    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw"]
Sep 17 19:07:34.177287 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:34.177260    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb638639_51b4_47b4_bcd2_43ce0e5872c4.slice/crio-663c4871047d18a50a044b6fa937ea2f02563b7a1700babd8d7577ad4468d158 WatchSource:0}: Error finding container 663c4871047d18a50a044b6fa937ea2f02563b7a1700babd8d7577ad4468d158: Status 404 returned error can't find the container with id 663c4871047d18a50a044b6fa937ea2f02563b7a1700babd8d7577ad4468d158
Sep 17 19:07:34.235486 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:34.235408    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-557d646d44-6wm5d"]
Sep 17 19:07:34.238688 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:34.238659    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc235293a_4120_48f2_8a31_21b5209b67a0.slice/crio-0d723fd3e74c12290f7c0182d89732a57166fe847b87cfa837e757eb73634cf9 WatchSource:0}: Error finding container 0d723fd3e74c12290f7c0182d89732a57166fe847b87cfa837e757eb73634cf9: Status 404 returned error can't find the container with id 0d723fd3e74c12290f7c0182d89732a57166fe847b87cfa837e757eb73634cf9
Sep 17 19:07:34.519516 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:34.519432    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-557d646d44-6wm5d" event={"ID":"c235293a-4120-48f2-8a31-21b5209b67a0","Type":"ContainerStarted","Data":"fb47ed6f7812d660c560cdb7f77abdf291a03646e27f87cb1b56890ecccf51f7"}
Sep 17 19:07:34.519516 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:34.519479    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-557d646d44-6wm5d" event={"ID":"c235293a-4120-48f2-8a31-21b5209b67a0","Type":"ContainerStarted","Data":"0d723fd3e74c12290f7c0182d89732a57166fe847b87cfa837e757eb73634cf9"}
Sep 17 19:07:34.519999 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:34.519551    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:34.520904 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:34.520873    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw" event={"ID":"bb638639-51b4-47b4-bcd2-43ce0e5872c4","Type":"ContainerStarted","Data":"663c4871047d18a50a044b6fa937ea2f02563b7a1700babd8d7577ad4468d158"}
Sep 17 19:07:34.555540 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:34.555486    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-557d646d44-6wm5d" podStartSLOduration=1.555467634 podStartE2EDuration="1.555467634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:34.554451076 +0000 UTC m=+77.480669502" watchObservedRunningTime="2026-09-17 19:07:34.555467634 +0000 UTC m=+77.481686111"
Sep 17 19:07:35.794619 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:35.794578    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:35.893690 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:35.893647    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:35.962225 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:35.962180    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:36.335636 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:36.335602    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:37.533700 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:37.533662    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw" event={"ID":"bb638639-51b4-47b4-bcd2-43ce0e5872c4","Type":"ContainerStarted","Data":"af93e38f83123bd1f71d617bfbcb4babd43b80f91de6ec37235b34028bba98a1"}
Sep 17 19:07:37.556590 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:37.556533    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59df778558-bnxjw" podStartSLOduration=2.0364739099999998 podStartE2EDuration="4.556518164s" totalImagesPullingTime="2.520044254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:37.55500361 +0000 UTC m=+80.481222034" watchObservedRunningTime="2026-09-17 19:07:37.556518164 +0000 UTC m=+80.482736586"
Sep 17 19:07:38.522381 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:38.522339    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:40.075505 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.075466    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:40.077628 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.077604    2567 scope.go:117] "RemoveContainer" containerID="55882f2db0912a86a74014202c396c46af6a359c551b8db3c92ef5b270f29e55"
Sep 17 19:07:40.077781 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:07:40.077762    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-2ztxw_openshift-console-operator(76e839db-c4d0-4f68-b9e6-3f8950193307)\"" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" podUID="76e839db-c4d0-4f68-b9e6-3f8950193307"
Sep 17 19:07:40.196430 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.196393    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:40.305792 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.305753    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:40.479426 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.479388    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lm4j6"]
Sep 17 19:07:40.484346 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.484326    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.488614 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.488587    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 19:07:40.488769 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.488737    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 19:07:40.489706 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.489677    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 19:07:40.489815 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.489720    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x7nzw\""
Sep 17 19:07:40.492327 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.492305    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 19:07:40.498050 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.498018    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:40.506611 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.506593    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sj5qz"
Sep 17 19:07:40.553114 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.553076    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m88v\" (UniqueName: \"kubernetes.io/projected/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-kube-api-access-5m88v\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.553290 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.553151    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-wtmp\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.553290 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.553232    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-tls\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.553290 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.553264    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-root\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.553405 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.553294    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-metrics-client-ca\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.553405 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.553325    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.553405 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.553342    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-accelerators-collector-config\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.553405 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.553383    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-textfile\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.553405 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.553401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-sys\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654330 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654292    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654491 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654340    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-accelerators-collector-config\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654491 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654378    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-textfile\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654491 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654403    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-sys\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654491 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654480    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m88v\" (UniqueName: \"kubernetes.io/projected/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-kube-api-access-5m88v\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654720 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654506    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-sys\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654720 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654524    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-wtmp\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654720 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654558    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-tls\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654720 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654622    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-root\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654720 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654648    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-metrics-client-ca\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654809    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-textfile\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654874    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-wtmp\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.654984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.654876    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-root\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.655283 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.655266    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-accelerators-collector-config\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.655418 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.655398    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-metrics-client-ca\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.656816 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.656795    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.656980 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.656962    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-node-exporter-tls\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.659835 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.659812    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:40.674426 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.674397    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m88v\" (UniqueName: \"kubernetes.io/projected/0de4d8a3-1226-4b73-a3d6-b52e3795f94e-kube-api-access-5m88v\") pod \"node-exporter-lm4j6\" (UID: \"0de4d8a3-1226-4b73-a3d6-b52e3795f94e\") " pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.793701 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.793609    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lm4j6"
Sep 17 19:07:40.805752 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:40.805717    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de4d8a3_1226_4b73_a3d6_b52e3795f94e.slice/crio-05c4903cccf59e42e5f12cfd10d64faae431526bdab9487f96f01510b02c5733 WatchSource:0}: Error finding container 05c4903cccf59e42e5f12cfd10d64faae431526bdab9487f96f01510b02c5733: Status 404 returned error can't find the container with id 05c4903cccf59e42e5f12cfd10d64faae431526bdab9487f96f01510b02c5733
Sep 17 19:07:40.881088 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:40.881049    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:41.112702 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:41.112659    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:41.251875 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:41.251834    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:41.353586 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:41.353545    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:41.458616 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:41.458528    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:41.545611 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:41.545506    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:41.549518 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:41.549489    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lm4j6" event={"ID":"0de4d8a3-1226-4b73-a3d6-b52e3795f94e","Type":"ContainerStarted","Data":"05c4903cccf59e42e5f12cfd10d64faae431526bdab9487f96f01510b02c5733"}
Sep 17 19:07:42.555377 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.555341    2567 generic.go:356] "Generic (PLEG): container finished" podID="0de4d8a3-1226-4b73-a3d6-b52e3795f94e" containerID="b6454243dc60aa574a0247807421637960cac6fdcbf5093eb8a75c018baf3a42" exitCode=0
Sep 17 19:07:42.555784 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.555416    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lm4j6" event={"ID":"0de4d8a3-1226-4b73-a3d6-b52e3795f94e","Type":"ContainerDied","Data":"b6454243dc60aa574a0247807421637960cac6fdcbf5093eb8a75c018baf3a42"}
Sep 17 19:07:42.578194 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.578165    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"]
Sep 17 19:07:42.582797 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.582773    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.586329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.586300    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 19:07:42.586471 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.586300    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 19:07:42.586471 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.586300    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 19:07:42.586735 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.586720    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 19:07:42.586735 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.586730    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e6pe6eqakba4p\""
Sep 17 19:07:42.587435 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.587418    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-trknj\""
Sep 17 19:07:42.587660 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.587646    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 19:07:42.597703 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.597674    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"]
Sep 17 19:07:42.673818 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.673780    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.673818 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.673821    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhd5\" (UniqueName: \"kubernetes.io/projected/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-kube-api-access-kxhd5\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.674036 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.673843    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-grpc-tls\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.674036 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.673859    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.674036 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.673935    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-tls\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.674036 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.673966    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-metrics-client-ca\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.674036 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.674008    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.674247 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.674045    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.774715 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.774668    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.774715 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.774720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.774995 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.774772    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.774995 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.774789    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhd5\" (UniqueName: \"kubernetes.io/projected/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-kube-api-access-kxhd5\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.774995 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.774815    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-grpc-tls\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.774995 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.774839    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.774995 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.774875    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-tls\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.774995 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.774898    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-metrics-client-ca\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.775722 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.775691    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-metrics-client-ca\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.777670 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.777637    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.777773 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.777690    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.777847 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.777795    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.778118 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.778097    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-grpc-tls\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.778211 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.778112    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.778256 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.778236    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-secret-thanos-querier-tls\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.784891 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.784860    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhd5\" (UniqueName: \"kubernetes.io/projected/83d76dee-ce50-4e39-a6a5-7e8e43c19fa2-kube-api-access-kxhd5\") pod \"thanos-querier-7c78b66bb-wqrwb\" (UID: \"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2\") " pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:42.929222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:42.929168    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:43.219626 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:43.219543    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"]
Sep 17 19:07:43.222509 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:43.222480    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d76dee_ce50_4e39_a6a5_7e8e43c19fa2.slice/crio-f8ac8036789ec05ba1c8b05f7432b16ffd97a99a14fe3c9581b84f54c95d725c WatchSource:0}: Error finding container f8ac8036789ec05ba1c8b05f7432b16ffd97a99a14fe3c9581b84f54c95d725c: Status 404 returned error can't find the container with id f8ac8036789ec05ba1c8b05f7432b16ffd97a99a14fe3c9581b84f54c95d725c
Sep 17 19:07:43.410415 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:43.410375    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:43.504262 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:43.504162    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:43.560402 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:43.560357    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb" event={"ID":"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2","Type":"ContainerStarted","Data":"f8ac8036789ec05ba1c8b05f7432b16ffd97a99a14fe3c9581b84f54c95d725c"}
Sep 17 19:07:43.562647 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:43.562623    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lm4j6" event={"ID":"0de4d8a3-1226-4b73-a3d6-b52e3795f94e","Type":"ContainerStarted","Data":"b669a5e8276941b6b52fa81d020f2fd52417e2c2cf9f46573e54d6e8deae78d5"}
Sep 17 19:07:43.562760 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:43.562665    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lm4j6" event={"ID":"0de4d8a3-1226-4b73-a3d6-b52e3795f94e","Type":"ContainerStarted","Data":"180aae667dc770d93bd9ce939db3b1e4574572b999c293f972a2bd5547ef92e8"}
Sep 17 19:07:43.569634 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:43.569612    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:43.584299 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:43.584250    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lm4j6" podStartSLOduration=2.5419268219999998 podStartE2EDuration="3.584234981s" totalImagesPullingTime="979.077335ms" totalInitContainerRuntime="63.230824ms" isStatefulPod=true podCreationTimestamp="2026-09-17 19:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:43.583473357 +0000 UTC m=+86.509691784" watchObservedRunningTime="2026-09-17 19:07:43.584234981 +0000 UTC m=+86.510453403"
Sep 17 19:07:43.807439 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:43.807341    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:43.908837 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:43.908804    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:45.059791 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.059688    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:45.103592 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.103568    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-676886d4d4-twqmc"]
Sep 17 19:07:45.107021 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.107002    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.114047 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.114019    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 19:07:45.114204 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.114056    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 19:07:45.115415 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.115039    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 19:07:45.115415 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.115359    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qjzzp\""
Sep 17 19:07:45.115415 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.115389    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 19:07:45.115572 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.115453    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-foht7t1tsj17r\""
Sep 17 19:07:45.120637 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.120611    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-676886d4d4-twqmc"]
Sep 17 19:07:45.194306 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.194207    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-client-ca-bundle\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.194470 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.194301    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f79bz\" (UniqueName: \"kubernetes.io/projected/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-kube-api-access-f79bz\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.194470 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.194337    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-secret-metrics-server-client-certs\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.194470 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.194362    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-secret-metrics-server-tls\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.194470 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.194379    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-audit-log\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.194470 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.194451    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-metrics-server-audit-profiles\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.194655 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.194585    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.296010 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.295969    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.296212 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.296040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-client-ca-bundle\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.296212 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.296086    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f79bz\" (UniqueName: \"kubernetes.io/projected/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-kube-api-access-f79bz\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.296337 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.296251    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-secret-metrics-server-client-certs\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.296337 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.296283    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-secret-metrics-server-tls\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.296337 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.296301    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-audit-log\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.296479 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.296343    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-metrics-server-audit-profiles\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.296866 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.296839    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-audit-log\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.296976 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.296877    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.297379 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.297359    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-metrics-server-audit-profiles\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.298876 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.298849    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-secret-metrics-server-client-certs\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.298986 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.298951    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-client-ca-bundle\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.299073 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.299053    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-secret-metrics-server-tls\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.304463 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.304438    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f79bz\" (UniqueName: \"kubernetes.io/projected/ff385afc-848d-4ac4-85fd-a1c8c9629ee5-kube-api-access-f79bz\") pod \"metrics-server-676886d4d4-twqmc\" (UID: \"ff385afc-848d-4ac4-85fd-a1c8c9629ee5\") " pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.393554 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.393513    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:45.418492 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.418459    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:07:45.492418 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.492383    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:45.573093 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.573058    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb" event={"ID":"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2","Type":"ContainerStarted","Data":"181663063a4f839679d004ab1f1a216c0c35ccbff1e8ebfa66a65f8102c5d8ab"}
Sep 17 19:07:45.573249 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.573170    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb" event={"ID":"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2","Type":"ContainerStarted","Data":"688b403069abd52946e8539fcde127d6e8c55526f5cbdb027fddbaf94d632b51"}
Sep 17 19:07:45.573249 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.573198    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb" event={"ID":"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2","Type":"ContainerStarted","Data":"85c0a3068134b207d545278d76551724dfc038c8b152ed7b1fdfcc304876e779"}
Sep 17 19:07:45.657380 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.657331    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"]
Sep 17 19:07:45.660966 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.660939    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.664848 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.664823    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 19:07:45.664998 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.664855    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 19:07:45.665467 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.665447    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 19:07:45.665726 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.665708    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 19:07:45.666955 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.666932    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 19:07:45.668242 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.668213    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jcwl9\""
Sep 17 19:07:45.671354 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.671332    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 19:07:45.689170 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.689126    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"]
Sep 17 19:07:45.698993 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.698951    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-676886d4d4-twqmc"]
Sep 17 19:07:45.702183 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:45.702108    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff385afc_848d_4ac4_85fd_a1c8c9629ee5.slice/crio-fdc33c688a3fdc52fd8265276f6e1f2ff8163e5de8d331fa11613812b9f0686d WatchSource:0}: Error finding container fdc33c688a3fdc52fd8265276f6e1f2ff8163e5de8d331fa11613812b9f0686d: Status 404 returned error can't find the container with id fdc33c688a3fdc52fd8265276f6e1f2ff8163e5de8d331fa11613812b9f0686d
Sep 17 19:07:45.800860 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.800763    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-telemeter-client-tls\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.800860 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.800819    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-serving-certs-ca-bundle\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.801102 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.800876    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.801102 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.800942    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m62bv\" (UniqueName: \"kubernetes.io/projected/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-kube-api-access-m62bv\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.801102 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.800991    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-secret-telemeter-client\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.801102 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.801040    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-metrics-client-ca\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.801102 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.801081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.801379 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.801188    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-federate-client-tls\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.894552 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.894525    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:45.902117 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.902091    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-federate-client-tls\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.902228 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.902129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-telemeter-client-tls\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.902228 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.902180    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-serving-certs-ca-bundle\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.902228 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.902204    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.902386 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.902232    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m62bv\" (UniqueName: \"kubernetes.io/projected/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-kube-api-access-m62bv\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.902386 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.902268    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-secret-telemeter-client\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.902386 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.902293    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-metrics-client-ca\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.902386 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.902327    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.903327 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.903064    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-serving-certs-ca-bundle\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.903327 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.903265    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-metrics-client-ca\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.903327 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.903265    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.904889 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.904857    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.905257 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.905231    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-secret-telemeter-client\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.905346 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.905239    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-federate-client-tls\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.905346 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.905320    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-telemeter-client-tls\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.911602 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.911581    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m62bv\" (UniqueName: \"kubernetes.io/projected/ca4dada6-f4d0-4f5d-b958-25c7a0f659ac-kube-api-access-m62bv\") pod \"telemeter-client-7fff9667cb-kkgh4\" (UID: \"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac\") " pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:45.973484 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:45.973462    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"
Sep 17 19:07:46.265451 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:46.265413    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fff9667cb-kkgh4"]
Sep 17 19:07:46.270809 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:46.270776    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4dada6_f4d0_4f5d_b958_25c7a0f659ac.slice/crio-3731e4d2b533fa0db3511478c3db9b5753208b036f16f49ef262111d3633de37 WatchSource:0}: Error finding container 3731e4d2b533fa0db3511478c3db9b5753208b036f16f49ef262111d3633de37: Status 404 returned error can't find the container with id 3731e4d2b533fa0db3511478c3db9b5753208b036f16f49ef262111d3633de37
Sep 17 19:07:46.581759 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:46.581723    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb" event={"ID":"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2","Type":"ContainerStarted","Data":"aec3ffce5871d17041b16e505d860b37d1328bc530f5ffd51ea98f14bb959666"}
Sep 17 19:07:46.581923 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:46.581805    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb" event={"ID":"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2","Type":"ContainerStarted","Data":"611795f58e40e76089cdf6fdc595b8d2bda989e88ec99a7c2e68c60f3d4d2050"}
Sep 17 19:07:46.581923 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:46.581832    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb" event={"ID":"83d76dee-ce50-4e39-a6a5-7e8e43c19fa2","Type":"ContainerStarted","Data":"01b6498ad9cb1238e5b7afd5d3d575a7259c20330711e8eb8c5e8717e1141f96"}
Sep 17 19:07:46.581923 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:46.581893    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:46.583094 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:46.583067    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-676886d4d4-twqmc" event={"ID":"ff385afc-848d-4ac4-85fd-a1c8c9629ee5","Type":"ContainerStarted","Data":"fdc33c688a3fdc52fd8265276f6e1f2ff8163e5de8d331fa11613812b9f0686d"}
Sep 17 19:07:46.584226 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:46.584205    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4" event={"ID":"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac","Type":"ContainerStarted","Data":"3731e4d2b533fa0db3511478c3db9b5753208b036f16f49ef262111d3633de37"}
Sep 17 19:07:46.767701 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:46.767661    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:47.596489 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:47.596432    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-676886d4d4-twqmc" event={"ID":"ff385afc-848d-4ac4-85fd-a1c8c9629ee5","Type":"ContainerStarted","Data":"200b0e4b0bc4b3b68848c1721e67eb39f19b0f805e85c26230d604a2778054a6"}
Sep 17 19:07:47.619402 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:47.619239    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb" podStartSLOduration=3.177367734 podStartE2EDuration="5.619222917s" totalImagesPullingTime="2.441855183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:46.61387191 +0000 UTC m=+89.540090338" watchObservedRunningTime="2026-09-17 19:07:47.619222917 +0000 UTC m=+90.545441339"
Sep 17 19:07:47.620360 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:47.620311    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-676886d4d4-twqmc" podStartSLOduration=1.337994677 podStartE2EDuration="2.620298958s" totalImagesPullingTime="1.282304281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:47.619028243 +0000 UTC m=+90.545246667" watchObservedRunningTime="2026-09-17 19:07:47.620298958 +0000 UTC m=+90.546517383"
Sep 17 19:07:49.606992 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:49.606963    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4" event={"ID":"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac","Type":"ContainerStarted","Data":"04b7098e13e9f86b8b3134db11a37d726e1fb275be80488f18a18e977d154143"}
Sep 17 19:07:50.612552 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:50.612507    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4" event={"ID":"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac","Type":"ContainerStarted","Data":"baa6484f90a46eccb4185f41b680060b1e224b68bbc29de0125451ca27e15bbd"}
Sep 17 19:07:50.612929 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:50.612608    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4" event={"ID":"ca4dada6-f4d0-4f5d-b958-25c7a0f659ac","Type":"ContainerStarted","Data":"c0cf9583279af0bf61df079be60d85283239d1510ba4ae0b6d3ff898cd6b6e46"}
Sep 17 19:07:50.641208 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:50.641124    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7fff9667cb-kkgh4" podStartSLOduration=2.435865933 podStartE2EDuration="5.641106023s" totalImagesPullingTime="3.20524009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:50.639216296 +0000 UTC m=+93.565434719" watchObservedRunningTime="2026-09-17 19:07:50.641106023 +0000 UTC m=+93.567324445"
Sep 17 19:07:51.078261 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:51.078180    2567 scope.go:117] "RemoveContainer" containerID="55882f2db0912a86a74014202c396c46af6a359c551b8db3c92ef5b270f29e55"
Sep 17 19:07:51.618830 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:51.618796    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:07:51.621474 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:51.619665    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" event={"ID":"76e839db-c4d0-4f68-b9e6-3f8950193307","Type":"ContainerStarted","Data":"5d7b4f4c73390d89876724af611b5d5374a2ff441d9953ae3549a1edd4d30cf1"}
Sep 17 19:07:51.621474 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:51.621464    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:07:51.646633 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:51.646576    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw" podStartSLOduration=51.363781831 podStartE2EDuration="53.646558179s" totalImagesPullingTime="2.282776348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:51.64388257 +0000 UTC m=+94.570100994" watchObservedRunningTime="2026-09-17 19:07:51.646558179 +0000 UTC m=+94.572776602"
Sep 17 19:07:52.212178 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.212131    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-2ztxw"
Sep 17 19:07:52.392795 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.392758    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-thnpp"]
Sep 17 19:07:52.396168 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.396125    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-thnpp"
Sep 17 19:07:52.398962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.398937    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 19:07:52.398962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.398957    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 19:07:52.399150 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.399015    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-96b2n\""
Sep 17 19:07:52.407831 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.407798    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:52.408217 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.408198    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-thnpp"]
Sep 17 19:07:52.502681 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.502582    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:52.566915 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.566872    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:52.570964 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.570937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk652\" (UniqueName: \"kubernetes.io/projected/0d51a30f-1aa8-4485-811c-2a7f0b5e9368-kube-api-access-fk652\") pod \"downloads-6594f4c75-thnpp\" (UID: \"0d51a30f-1aa8-4485-811c-2a7f0b5e9368\") " pod="openshift-console/downloads-6594f4c75-thnpp"
Sep 17 19:07:52.604863 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.604836    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c78b66bb-wqrwb"
Sep 17 19:07:52.671896 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.671862    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk652\" (UniqueName: \"kubernetes.io/projected/0d51a30f-1aa8-4485-811c-2a7f0b5e9368-kube-api-access-fk652\") pod \"downloads-6594f4c75-thnpp\" (UID: \"0d51a30f-1aa8-4485-811c-2a7f0b5e9368\") " pod="openshift-console/downloads-6594f4c75-thnpp"
Sep 17 19:07:52.688352 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.688316    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk652\" (UniqueName: \"kubernetes.io/projected/0d51a30f-1aa8-4485-811c-2a7f0b5e9368-kube-api-access-fk652\") pod \"downloads-6594f4c75-thnpp\" (UID: \"0d51a30f-1aa8-4485-811c-2a7f0b5e9368\") " pod="openshift-console/downloads-6594f4c75-thnpp"
Sep 17 19:07:52.706155 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.706114    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-thnpp"
Sep 17 19:07:52.718600 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.718569    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:52.993732 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:52.993710    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-thnpp"]
Sep 17 19:07:52.995975 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:52.995946    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d51a30f_1aa8_4485_811c_2a7f0b5e9368.slice/crio-145348caed2a0aa0812e7a1335a402d29ab631779694f4247ade9de4a519c26c WatchSource:0}: Error finding container 145348caed2a0aa0812e7a1335a402d29ab631779694f4247ade9de4a519c26c: Status 404 returned error can't find the container with id 145348caed2a0aa0812e7a1335a402d29ab631779694f4247ade9de4a519c26c
Sep 17 19:07:53.078663 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:53.078575    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:53.178427 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:53.178390    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:53.508244 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:53.508196    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:53.603510 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:53.603459    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:53.629362 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:53.629317    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-thnpp" event={"ID":"0d51a30f-1aa8-4485-811c-2a7f0b5e9368","Type":"ContainerStarted","Data":"145348caed2a0aa0812e7a1335a402d29ab631779694f4247ade9de4a519c26c"}
Sep 17 19:07:53.704272 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:53.704233    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:53.804025 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:53.803885    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:53.927804 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:53.927763    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:54.004477 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:54.004442    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:54.104535 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:54.104436    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:54.289608 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:54.289573    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:54.503788 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:54.503744    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:54.786213 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:54.786099    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:54.857339 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:54.857301    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:55.530921 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:55.530884    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-557d646d44-6wm5d"
Sep 17 19:07:55.718710 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:55.718647    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:55.819979 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:55.819886    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:55.922044 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:55.922009    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:55.999044 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:55.999002    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:57.223155 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:57.223090    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:57.323925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:57.323868    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:58.412900 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.412865    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d9f58cd46-spghn"]
Sep 17 19:07:58.419334 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.419307    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.422674 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.422644    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 19:07:58.422674 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.422647    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 19:07:58.423843 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.423820    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 19:07:58.424084 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.423967    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 19:07:58.424084 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.423846    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rlcgb\""
Sep 17 19:07:58.424084 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.423940    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 19:07:58.432505 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.432480    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9f58cd46-spghn"]
Sep 17 19:07:58.516590 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.516553    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-service-ca\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.516861 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.516617    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-serving-cert\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.516861 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.516676    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl22t\" (UniqueName: \"kubernetes.io/projected/8821bb70-5c96-4155-ab73-0a7445cfe1d9-kube-api-access-cl22t\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.516861 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.516706    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-oauth-serving-cert\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.516861 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.516752    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-config\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.516861 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.516779    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-oauth-config\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.617671 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.617633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-serving-cert\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.617875 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.617690    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl22t\" (UniqueName: \"kubernetes.io/projected/8821bb70-5c96-4155-ab73-0a7445cfe1d9-kube-api-access-cl22t\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.617875 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.617728    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-oauth-serving-cert\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.617875 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.617780    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-config\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.617875 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.617805    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-oauth-config\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.617875 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.617840    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-service-ca\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.618676 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.618645    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-service-ca\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.618808 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.618728    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-config\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.618808 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.618729    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-oauth-serving-cert\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.620697 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.620670    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-serving-cert\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.620824 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.620704    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-oauth-config\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.627638 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.627610    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl22t\" (UniqueName: \"kubernetes.io/projected/8821bb70-5c96-4155-ab73-0a7445cfe1d9-kube-api-access-cl22t\") pod \"console-6d9f58cd46-spghn\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") " pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:58.730600 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:58.730504    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:07:59.028589 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:59.028509    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9f58cd46-spghn"]
Sep 17 19:07:59.031691 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:07:59.031659    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8821bb70_5c96_4155_ab73_0a7445cfe1d9.slice/crio-503c3843c4514201aaf4f5a123e1a59b710557aed3f49a9c7e088e1f06d7f7e5 WatchSource:0}: Error finding container 503c3843c4514201aaf4f5a123e1a59b710557aed3f49a9c7e088e1f06d7f7e5: Status 404 returned error can't find the container with id 503c3843c4514201aaf4f5a123e1a59b710557aed3f49a9c7e088e1f06d7f7e5
Sep 17 19:07:59.665065 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:07:59.665026    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9f58cd46-spghn" event={"ID":"8821bb70-5c96-4155-ab73-0a7445cfe1d9","Type":"ContainerStarted","Data":"503c3843c4514201aaf4f5a123e1a59b710557aed3f49a9c7e088e1f06d7f7e5"}
Sep 17 19:08:00.551887 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:00.551843    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:00.661095 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:00.661004    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:01.506812 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:01.506782    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mshnf"
Sep 17 19:08:02.109945 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:02.109902    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:02.646242 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:02.646213    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:02.712917 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:02.712873    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:04.730515 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:04.730477    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:04.807550 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:04.807512    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:05.419543 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:05.419495    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:08:05.419728 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:05.419579    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:08:08.518814 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:08.518773    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:08.620479 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:08.620429    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:09.121062 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:09.121020    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:09.222210 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:09.222161    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:09.284895 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:09.284852    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:11.723974 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:11.723926    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9f58cd46-spghn" event={"ID":"8821bb70-5c96-4155-ab73-0a7445cfe1d9","Type":"ContainerStarted","Data":"040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a"}
Sep 17 19:08:11.725996 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:11.725955    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-thnpp" event={"ID":"0d51a30f-1aa8-4485-811c-2a7f0b5e9368","Type":"ContainerStarted","Data":"1c7c07657000496de925364d7a6d862d4a40224c29e329f7b9546d6a5a31f982"}
Sep 17 19:08:11.726215 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:11.726177    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-thnpp"
Sep 17 19:08:11.745933 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:11.745870    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d9f58cd46-spghn" podStartSLOduration=1.824284262 podStartE2EDuration="13.745850854s" totalImagesPullingTime="11.921566592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:11.743924712 +0000 UTC m=+114.670143148" watchObservedRunningTime="2026-09-17 19:08:11.745850854 +0000 UTC m=+114.672069276"
Sep 17 19:08:11.746991 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:11.746966    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-thnpp"
Sep 17 19:08:11.766519 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:11.766456    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-thnpp" podStartSLOduration=1.773369911 podStartE2EDuration="19.766438425s" totalImagesPullingTime="17.993068514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:11.765482743 +0000 UTC m=+114.691701169" watchObservedRunningTime="2026-09-17 19:08:11.766438425 +0000 UTC m=+114.692656848"
Sep 17 19:08:12.927008 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:12.926963    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:13.027564 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:13.027520    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:15.494880 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:15.494829    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:15.569519 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:15.569476    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:16.523863 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:16.523814    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:16.633677 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:16.633638    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:17.755791 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:17.755752    2567 generic.go:356] "Generic (PLEG): container finished" podID="da661868-318a-4f63-8068-f57f60e3f5e7" containerID="f89f0ab359dd97b739136951c4d942e95e70094b8192d51d0a60842b9e84339b" exitCode=0
Sep 17 19:08:17.756297 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:17.755832    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7" event={"ID":"da661868-318a-4f63-8068-f57f60e3f5e7","Type":"ContainerDied","Data":"f89f0ab359dd97b739136951c4d942e95e70094b8192d51d0a60842b9e84339b"}
Sep 17 19:08:17.756367 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:17.756328    2567 scope.go:117] "RemoveContainer" containerID="f89f0ab359dd97b739136951c4d942e95e70094b8192d51d0a60842b9e84339b"
Sep 17 19:08:18.730959 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:18.730912    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:08:18.731182 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:18.730987    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:08:18.737158 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:18.737117    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:08:18.770469 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:18.770424    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-kj7p7" event={"ID":"da661868-318a-4f63-8068-f57f60e3f5e7","Type":"ContainerStarted","Data":"2c38a7c23609109a5b5a91e96ce58ff6a14a69eaff2d21c098fc93ff08c085f0"}
Sep 17 19:08:18.775359 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:18.775332    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:08:19.431113 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:19.431076    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9f58cd46-spghn"]
Sep 17 19:08:19.580186 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:19.580127    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-97zqg_c653f4cb-bc89-4470-9d85-f6550ff7f325/cluster-monitoring-operator/0.log"
Sep 17 19:08:20.377492 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:20.377441    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-676886d4d4-twqmc_ff385afc-848d-4ac4-85fd-a1c8c9629ee5/metrics-server/0.log"
Sep 17 19:08:20.777696 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:20.777620    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lm4j6_0de4d8a3-1226-4b73-a3d6-b52e3795f94e/init-textfile/0.log"
Sep 17 19:08:20.980033 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:20.979995    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lm4j6_0de4d8a3-1226-4b73-a3d6-b52e3795f94e/node-exporter/0.log"
Sep 17 19:08:21.180048 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:21.180018    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lm4j6_0de4d8a3-1226-4b73-a3d6-b52e3795f94e/kube-rbac-proxy/0.log"
Sep 17 19:08:23.317946 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:23.317903    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:23.429790 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:23.429754    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:23.916469 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:23.916434    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:23.980443 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:23.980406    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:25.177892 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:25.177865    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fff9667cb-kkgh4_ca4dada6-f4d0-4f5d-b958-25c7a0f659ac/telemeter-client/0.log"
Sep 17 19:08:25.377246 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:25.377218    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fff9667cb-kkgh4_ca4dada6-f4d0-4f5d-b958-25c7a0f659ac/reload/0.log"
Sep 17 19:08:25.424891 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:25.424860    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:08:25.428848 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:25.428714    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-676886d4d4-twqmc"
Sep 17 19:08:25.578501 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:25.578471    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fff9667cb-kkgh4_ca4dada6-f4d0-4f5d-b958-25c7a0f659ac/kube-rbac-proxy/0.log"
Sep 17 19:08:25.778643 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:25.778562    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/thanos-query/0.log"
Sep 17 19:08:25.977316 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:25.977290    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/kube-rbac-proxy-web/0.log"
Sep 17 19:08:26.177541 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:26.177489    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/kube-rbac-proxy/0.log"
Sep 17 19:08:26.377261 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:26.377227    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/prom-label-proxy/0.log"
Sep 17 19:08:26.576603 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:26.576500    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/kube-rbac-proxy-rules/0.log"
Sep 17 19:08:26.776993 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:26.776965    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/kube-rbac-proxy-metrics/0.log"
Sep 17 19:08:27.177669 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:27.177636    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:08:27.380905 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:27.380850    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/3.log"
Sep 17 19:08:27.578312 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:27.578204    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d9f58cd46-spghn_8821bb70-5c96-4155-ab73-0a7445cfe1d9/console/0.log"
Sep 17 19:08:27.979995 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:27.979963    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-thnpp_0d51a30f-1aa8-4485-811c-2a7f0b5e9368/download-server/0.log"
Sep 17 19:08:30.507287 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:30.507250    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:30.587244 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:30.587108    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:45.799096 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:45.799016    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d9f58cd46-spghn" podUID="8821bb70-5c96-4155-ab73-0a7445cfe1d9" containerName="console" containerID="cri-o://040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a" gracePeriod=15
Sep 17 19:08:46.096670 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.096633    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:08:46.256758 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.256716    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl22t\" (UniqueName: \"kubernetes.io/projected/8821bb70-5c96-4155-ab73-0a7445cfe1d9-kube-api-access-cl22t\") pod \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") "
Sep 17 19:08:46.256973 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.256771    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-service-ca\") pod \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") "
Sep 17 19:08:46.256973 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.256807    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-oauth-serving-cert\") pod \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") "
Sep 17 19:08:46.256973 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.256846    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-serving-cert\") pod \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") "
Sep 17 19:08:46.256973 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.256888    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-config\") pod \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") "
Sep 17 19:08:46.256973 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.256924    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-oauth-config\") pod \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\" (UID: \"8821bb70-5c96-4155-ab73-0a7445cfe1d9\") "
Sep 17 19:08:46.257331 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.257307    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8821bb70-5c96-4155-ab73-0a7445cfe1d9" (UID: "8821bb70-5c96-4155-ab73-0a7445cfe1d9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:46.257385 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.257331    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-service-ca" (OuterVolumeSpecName: "service-ca") pod "8821bb70-5c96-4155-ab73-0a7445cfe1d9" (UID: "8821bb70-5c96-4155-ab73-0a7445cfe1d9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:46.257422 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.257374    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-config" (OuterVolumeSpecName: "console-config") pod "8821bb70-5c96-4155-ab73-0a7445cfe1d9" (UID: "8821bb70-5c96-4155-ab73-0a7445cfe1d9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:46.259236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.259203    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8821bb70-5c96-4155-ab73-0a7445cfe1d9" (UID: "8821bb70-5c96-4155-ab73-0a7445cfe1d9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:46.259236 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.259225    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8821bb70-5c96-4155-ab73-0a7445cfe1d9-kube-api-access-cl22t" (OuterVolumeSpecName: "kube-api-access-cl22t") pod "8821bb70-5c96-4155-ab73-0a7445cfe1d9" (UID: "8821bb70-5c96-4155-ab73-0a7445cfe1d9"). InnerVolumeSpecName "kube-api-access-cl22t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:46.259378 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.259354    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8821bb70-5c96-4155-ab73-0a7445cfe1d9" (UID: "8821bb70-5c96-4155-ab73-0a7445cfe1d9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:46.358288 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.358238    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl22t\" (UniqueName: \"kubernetes.io/projected/8821bb70-5c96-4155-ab73-0a7445cfe1d9-kube-api-access-cl22t\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.358288 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.358286    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-service-ca\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.358500 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.358299    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-oauth-serving-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.358500 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.358312    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-serving-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.358500 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.358324    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.358500 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.358335    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8821bb70-5c96-4155-ab73-0a7445cfe1d9-console-oauth-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:08:46.884148 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.884093    2567 generic.go:356] "Generic (PLEG): container finished" podID="8821bb70-5c96-4155-ab73-0a7445cfe1d9" containerID="040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a" exitCode=0
Sep 17 19:08:46.884654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.884169    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9f58cd46-spghn"
Sep 17 19:08:46.884654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.884185    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9f58cd46-spghn" event={"ID":"8821bb70-5c96-4155-ab73-0a7445cfe1d9","Type":"ContainerDied","Data":"040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a"}
Sep 17 19:08:46.884654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.884249    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9f58cd46-spghn" event={"ID":"8821bb70-5c96-4155-ab73-0a7445cfe1d9","Type":"ContainerDied","Data":"503c3843c4514201aaf4f5a123e1a59b710557aed3f49a9c7e088e1f06d7f7e5"}
Sep 17 19:08:46.884654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.884273    2567 scope.go:117] "RemoveContainer" containerID="040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a"
Sep 17 19:08:46.900808 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.900787    2567 scope.go:117] "RemoveContainer" containerID="040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a"
Sep 17 19:08:46.901200 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:08:46.901175    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a\": container with ID starting with 040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a not found: ID does not exist" containerID="040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a"
Sep 17 19:08:46.901285 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.901213    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a"} err="failed to get container status \"040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a\": rpc error: code = NotFound desc = could not find container \"040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a\": container with ID starting with 040cfaad7f30a130026bba5481ea7b4eeed453f8689d1fbc69e8c162cd00be3a not found: ID does not exist"
Sep 17 19:08:46.907573 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.907540    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9f58cd46-spghn"]
Sep 17 19:08:46.912192 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:46.912164    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d9f58cd46-spghn"]
Sep 17 19:08:47.082977 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:47.082943    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8821bb70-5c96-4155-ab73-0a7445cfe1d9" path="/var/lib/kubelet/pods/8821bb70-5c96-4155-ab73-0a7445cfe1d9/volumes"
Sep 17 19:08:56.624525 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:56.624482    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:56.690936 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:56.690873    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:59.244496 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:08:59.244459    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:08.620562 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:08.620481    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:08.710191 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:08.710158    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:15.495513 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:15.495468    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:15.594387 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:15.594353    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:29.876476 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.876443    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74c6586659-vml5x"]
Sep 17 19:09:29.876878 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.876767    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8821bb70-5c96-4155-ab73-0a7445cfe1d9" containerName="console"
Sep 17 19:09:29.876878 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.876779    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8821bb70-5c96-4155-ab73-0a7445cfe1d9" containerName="console"
Sep 17 19:09:29.876878 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.876839    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8821bb70-5c96-4155-ab73-0a7445cfe1d9" containerName="console"
Sep 17 19:09:29.881372 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.881351    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:29.884222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.884197    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 19:09:29.885362 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.885330    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rlcgb\""
Sep 17 19:09:29.885496 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.885364    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 19:09:29.885496 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.885364    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 19:09:29.885637 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.885612    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 19:09:29.885922 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.885899    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 19:09:29.890536 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.890514    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 19:09:29.890632 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.890514    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c6586659-vml5x"]
Sep 17 19:09:29.922980 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:29.922935    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:30.023641 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.023605    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:30.038667 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.038631    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-serving-cert\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.038853 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.038680    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-trusted-ca-bundle\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.038853 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.038701    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-oauth-config\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.038853 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.038724    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-service-ca\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.038853 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.038783    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-oauth-serving-cert\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.038853 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.038820    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-console-config\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.044058 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.038848    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6s7\" (UniqueName: \"kubernetes.io/projected/190002cb-b170-4498-86ab-3b37fec3cc8a-kube-api-access-tm6s7\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.121893 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.121852    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:30.140031 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.139943    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-serving-cert\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.140031 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.139999    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-trusted-ca-bundle\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.140031 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.140029    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-oauth-config\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.140335 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.140045    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-service-ca\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.140335 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.140079    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-oauth-serving-cert\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.140335 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.140278    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-console-config\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.140483 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.140355    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6s7\" (UniqueName: \"kubernetes.io/projected/190002cb-b170-4498-86ab-3b37fec3cc8a-kube-api-access-tm6s7\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.140968 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.140942    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-trusted-ca-bundle\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.141088 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.140952    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-service-ca\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.141088 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.140961    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-oauth-serving-cert\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.141088 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.140965    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-console-config\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.142637 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.142612    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-oauth-config\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.142750 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.142721    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-serving-cert\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.149913 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.149885    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6s7\" (UniqueName: \"kubernetes.io/projected/190002cb-b170-4498-86ab-3b37fec3cc8a-kube-api-access-tm6s7\") pod \"console-74c6586659-vml5x\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") " pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.187132 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.187087    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:30.193019 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.192981    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:30.465188 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:30.465157    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c6586659-vml5x"]
Sep 17 19:09:30.467776 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:09:30.467734    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190002cb_b170_4498_86ab_3b37fec3cc8a.slice/crio-e51a14dae08e798e80b02807fd440e68da137c27a42d8dbbb72d3696a57ffed3 WatchSource:0}: Error finding container e51a14dae08e798e80b02807fd440e68da137c27a42d8dbbb72d3696a57ffed3: Status 404 returned error can't find the container with id e51a14dae08e798e80b02807fd440e68da137c27a42d8dbbb72d3696a57ffed3
Sep 17 19:09:31.061046 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:31.061011    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c6586659-vml5x" event={"ID":"190002cb-b170-4498-86ab-3b37fec3cc8a","Type":"ContainerStarted","Data":"65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056"}
Sep 17 19:09:31.061480 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:31.061061    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c6586659-vml5x" event={"ID":"190002cb-b170-4498-86ab-3b37fec3cc8a","Type":"ContainerStarted","Data":"e51a14dae08e798e80b02807fd440e68da137c27a42d8dbbb72d3696a57ffed3"}
Sep 17 19:09:31.082499 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:31.082400    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74c6586659-vml5x" podStartSLOduration=2.08237579 podStartE2EDuration="2.08237579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:31.081643328 +0000 UTC m=+194.007861754" watchObservedRunningTime="2026-09-17 19:09:31.08237579 +0000 UTC m=+194.008594227"
Sep 17 19:09:40.193774 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:40.193725    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:40.194227 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:40.193805    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:40.198798 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:40.198774    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:41.102580 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:41.102550    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:09:41.290486 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:41.290446    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:41.355002 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:41.354909    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:41.412441 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:41.412407    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:48.676176 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:48.676120    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:48.775208 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:09:48.775162    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:07.938609 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:07.938569    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:08.036341 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:08.036308    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:08.101364 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:08.101333    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:30.517316 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:30.517278    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:30.660099 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:30.660062    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:36.153717 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:36.153627    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:36.434279 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:36.434191    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:36.621402 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:36.621365    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:36.783469 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:36.783384    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:37.001811 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.001774    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"]
Sep 17 19:10:37.005039 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.005022    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:37.007686 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.007657    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 19:10:37.007819 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.007689    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 19:10:37.007819 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.007790    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 19:10:37.007946 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.007930    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 19:10:37.008958 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.008932    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-v9x2p\""
Sep 17 19:10:37.009013 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.008965    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 19:10:37.015378 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.015353    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"]
Sep 17 19:10:37.108917 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.108886    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/076183bb-c089-45e4-9e40-a57ebb44af19-certificates\") pod \"keda-metrics-apiserver-77857b7b98-qdqld\" (UID: \"076183bb-c089-45e4-9e40-a57ebb44af19\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:37.109082 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.108935    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/076183bb-c089-45e4-9e40-a57ebb44af19-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-qdqld\" (UID: \"076183bb-c089-45e4-9e40-a57ebb44af19\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:37.109082 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.108998    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscnq\" (UniqueName: \"kubernetes.io/projected/076183bb-c089-45e4-9e40-a57ebb44af19-kube-api-access-nscnq\") pod \"keda-metrics-apiserver-77857b7b98-qdqld\" (UID: \"076183bb-c089-45e4-9e40-a57ebb44af19\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:37.210620 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.210580    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/076183bb-c089-45e4-9e40-a57ebb44af19-certificates\") pod \"keda-metrics-apiserver-77857b7b98-qdqld\" (UID: \"076183bb-c089-45e4-9e40-a57ebb44af19\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:37.211092 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.210665    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/076183bb-c089-45e4-9e40-a57ebb44af19-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-qdqld\" (UID: \"076183bb-c089-45e4-9e40-a57ebb44af19\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:37.211092 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:10:37.210687    2567 secret.go:281] references non-existent secret key: tls.crt
Sep 17 19:10:37.211092 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:10:37.210704    2567 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 17 19:10:37.211092 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.210707    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscnq\" (UniqueName: \"kubernetes.io/projected/076183bb-c089-45e4-9e40-a57ebb44af19-kube-api-access-nscnq\") pod \"keda-metrics-apiserver-77857b7b98-qdqld\" (UID: \"076183bb-c089-45e4-9e40-a57ebb44af19\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:37.211092 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:10:37.210728    2567 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld: references non-existent secret key: tls.crt
Sep 17 19:10:37.211092 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:10:37.210802    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/076183bb-c089-45e4-9e40-a57ebb44af19-certificates podName:076183bb-c089-45e4-9e40-a57ebb44af19 nodeName:}" failed. No retries permitted until 2026-09-17 19:10:37.710779354 +0000 UTC m=+260.636997769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/076183bb-c089-45e4-9e40-a57ebb44af19-certificates") pod "keda-metrics-apiserver-77857b7b98-qdqld" (UID: "076183bb-c089-45e4-9e40-a57ebb44af19") : references non-existent secret key: tls.crt
Sep 17 19:10:37.211092 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.210980    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/076183bb-c089-45e4-9e40-a57ebb44af19-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-qdqld\" (UID: \"076183bb-c089-45e4-9e40-a57ebb44af19\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:37.219247 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.219216    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:37.220693 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.220662    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscnq\" (UniqueName: \"kubernetes.io/projected/076183bb-c089-45e4-9e40-a57ebb44af19-kube-api-access-nscnq\") pod \"keda-metrics-apiserver-77857b7b98-qdqld\" (UID: \"076183bb-c089-45e4-9e40-a57ebb44af19\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:37.493302 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.493217    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:37.509752 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.509720    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zcm6q"]
Sep 17 19:10:37.513052 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.513029    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-zcm6q"
Sep 17 19:10:37.513391 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.513369    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt6ts\" (UniqueName: \"kubernetes.io/projected/c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b-kube-api-access-mt6ts\") pod \"keda-admission-6c4d864b86-zcm6q\" (UID: \"c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b\") " pod="openshift-keda/keda-admission-6c4d864b86-zcm6q"
Sep 17 19:10:37.513448 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.513416    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b-certificates\") pod \"keda-admission-6c4d864b86-zcm6q\" (UID: \"c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b\") " pod="openshift-keda/keda-admission-6c4d864b86-zcm6q"
Sep 17 19:10:37.515592 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.515577    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 19:10:37.520808 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.520784    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zcm6q"]
Sep 17 19:10:37.590395 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.590360    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:37.619799 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.619731    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt6ts\" (UniqueName: \"kubernetes.io/projected/c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b-kube-api-access-mt6ts\") pod \"keda-admission-6c4d864b86-zcm6q\" (UID: \"c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b\") " pod="openshift-keda/keda-admission-6c4d864b86-zcm6q"
Sep 17 19:10:37.620007 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.619840    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b-certificates\") pod \"keda-admission-6c4d864b86-zcm6q\" (UID: \"c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b\") " pod="openshift-keda/keda-admission-6c4d864b86-zcm6q"
Sep 17 19:10:37.622916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.622886    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b-certificates\") pod \"keda-admission-6c4d864b86-zcm6q\" (UID: \"c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b\") " pod="openshift-keda/keda-admission-6c4d864b86-zcm6q"
Sep 17 19:10:37.633293 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.633261    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt6ts\" (UniqueName: \"kubernetes.io/projected/c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b-kube-api-access-mt6ts\") pod \"keda-admission-6c4d864b86-zcm6q\" (UID: \"c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b\") " pod="openshift-keda/keda-admission-6c4d864b86-zcm6q"
Sep 17 19:10:37.694246 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.694204    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:37.720403 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.720363    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/076183bb-c089-45e4-9e40-a57ebb44af19-certificates\") pod \"keda-metrics-apiserver-77857b7b98-qdqld\" (UID: \"076183bb-c089-45e4-9e40-a57ebb44af19\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:37.722984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.722952    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/076183bb-c089-45e4-9e40-a57ebb44af19-certificates\") pod \"keda-metrics-apiserver-77857b7b98-qdqld\" (UID: \"076183bb-c089-45e4-9e40-a57ebb44af19\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:37.824597 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.824504    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-zcm6q"
Sep 17 19:10:37.891206 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.891127    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:37.916305 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:37.916270    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:38.013462 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:38.013432    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:38.096177 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:38.096153    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-zcm6q"]
Sep 17 19:10:38.098254 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:10:38.098227    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94546fa_a8bb_4db5_8e4c_0cb96a2ba25b.slice/crio-557e152593ca19a93ae7f24c08b662e63add26fcee3df3ee06585772813eacca WatchSource:0}: Error finding container 557e152593ca19a93ae7f24c08b662e63add26fcee3df3ee06585772813eacca: Status 404 returned error can't find the container with id 557e152593ca19a93ae7f24c08b662e63add26fcee3df3ee06585772813eacca
Sep 17 19:10:38.168615 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:38.168587    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:38.190370 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:38.190347    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"]
Sep 17 19:10:38.192758 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:10:38.192730    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076183bb_c089_45e4_9e40_a57ebb44af19.slice/crio-f918ab228a2523d01c35f4e1e14aa7cd356d41114d5a7f8def4145c21526e045 WatchSource:0}: Error finding container f918ab228a2523d01c35f4e1e14aa7cd356d41114d5a7f8def4145c21526e045: Status 404 returned error can't find the container with id f918ab228a2523d01c35f4e1e14aa7cd356d41114d5a7f8def4145c21526e045
Sep 17 19:10:38.264888 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:38.264850    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:38.321260 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:38.321222    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld" event={"ID":"076183bb-c089-45e4-9e40-a57ebb44af19","Type":"ContainerStarted","Data":"f918ab228a2523d01c35f4e1e14aa7cd356d41114d5a7f8def4145c21526e045"}
Sep 17 19:10:38.322439 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:38.322410    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-zcm6q" event={"ID":"c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b","Type":"ContainerStarted","Data":"557e152593ca19a93ae7f24c08b662e63add26fcee3df3ee06585772813eacca"}
Sep 17 19:10:42.345064 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:42.345024    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld" event={"ID":"076183bb-c089-45e4-9e40-a57ebb44af19","Type":"ContainerStarted","Data":"a02b40a56d815e6ec1403e4922af265ec1ffd61bbb0ee6fae546f5b0d813fe9e"}
Sep 17 19:10:42.345514 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:42.345123    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:10:42.346996 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:42.346967    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-zcm6q" event={"ID":"c94546fa-a8bb-4db5-8e4c-0cb96a2ba25b","Type":"ContainerStarted","Data":"bf72f9e9673a50ebc3886dda92c6319c55b542f1a3a64d13007796192354e5e8"}
Sep 17 19:10:42.347149 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:42.347028    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-zcm6q"
Sep 17 19:10:42.384013 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:42.383947    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld" podStartSLOduration=3.006732819 podStartE2EDuration="6.383926747s" totalImagesPullingTime="3.377193928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:42.382423357 +0000 UTC m=+265.308641780" watchObservedRunningTime="2026-09-17 19:10:42.383926747 +0000 UTC m=+265.310145170"
Sep 17 19:10:42.407865 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:42.407809    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-zcm6q" podStartSLOduration=1.9455252189999999 podStartE2EDuration="5.407790625s" totalImagesPullingTime="3.462265406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:10:42.406548968 +0000 UTC m=+265.332767392" watchObservedRunningTime="2026-09-17 19:10:42.407790625 +0000 UTC m=+265.334009048"
Sep 17 19:10:42.941887 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:42.941846    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:46.486485 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:46.486443    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:46.549763 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:46.549717    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:53.358064 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:10:53.358033    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qdqld"
Sep 17 19:11:03.354747 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:03.354718    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-zcm6q"
Sep 17 19:11:18.980753 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:18.980721    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:11:18.985494 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:18.985471    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:11:18.985681 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:18.985597    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:11:18.990028 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:18.990009    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:11:18.991551 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:18.991534    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 19:11:29.189830 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:29.189794    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:30.327953 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:30.327917    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:46.339659 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:46.339623    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:57.770611 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.770561    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:57.848304 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.848271    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"]
Sep 17 19:11:57.851821 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.851778    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"
Sep 17 19:11:57.852001 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.851976    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:57.856053 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.856030    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 19:11:57.856213 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.856081    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 19:11:57.856449 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.856436    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 19:11:57.857398 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.857374    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-bc6zn\""
Sep 17 19:11:57.865594 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.865565    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"]
Sep 17 19:11:57.943113 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.943074    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9mgk\" (UniqueName: \"kubernetes.io/projected/b491aad8-785b-4e38-b7d3-d4bafdb77743-kube-api-access-b9mgk\") pod \"llmisvc-controller-manager-645f9c8fd6-9bzlb\" (UID: \"b491aad8-785b-4e38-b7d3-d4bafdb77743\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"
Sep 17 19:11:57.943320 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.943201    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b491aad8-785b-4e38-b7d3-d4bafdb77743-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-9bzlb\" (UID: \"b491aad8-785b-4e38-b7d3-d4bafdb77743\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"
Sep 17 19:11:57.995179 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:57.995130    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:58.044637 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:58.044553    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b491aad8-785b-4e38-b7d3-d4bafdb77743-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-9bzlb\" (UID: \"b491aad8-785b-4e38-b7d3-d4bafdb77743\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"
Sep 17 19:11:58.044637 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:58.044621    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9mgk\" (UniqueName: \"kubernetes.io/projected/b491aad8-785b-4e38-b7d3-d4bafdb77743-kube-api-access-b9mgk\") pod \"llmisvc-controller-manager-645f9c8fd6-9bzlb\" (UID: \"b491aad8-785b-4e38-b7d3-d4bafdb77743\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"
Sep 17 19:11:58.047105 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:58.047081    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b491aad8-785b-4e38-b7d3-d4bafdb77743-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-9bzlb\" (UID: \"b491aad8-785b-4e38-b7d3-d4bafdb77743\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"
Sep 17 19:11:58.053153 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:58.053115    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9mgk\" (UniqueName: \"kubernetes.io/projected/b491aad8-785b-4e38-b7d3-d4bafdb77743-kube-api-access-b9mgk\") pod \"llmisvc-controller-manager-645f9c8fd6-9bzlb\" (UID: \"b491aad8-785b-4e38-b7d3-d4bafdb77743\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"
Sep 17 19:11:58.070374 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:58.070339    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:58.162739 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:58.162701    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"
Sep 17 19:11:58.438037 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:58.438004    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"]
Sep 17 19:11:58.441193 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:11:58.441128    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb491aad8_785b_4e38_b7d3_d4bafdb77743.slice/crio-e5c8880fa577fca1c713fcde89b8173f855fc18f9f0fe46f97041dc5c0748a5d WatchSource:0}: Error finding container e5c8880fa577fca1c713fcde89b8173f855fc18f9f0fe46f97041dc5c0748a5d: Status 404 returned error can't find the container with id e5c8880fa577fca1c713fcde89b8173f855fc18f9f0fe46f97041dc5c0748a5d
Sep 17 19:11:58.443105 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:58.443082    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:11:58.665921 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:58.665887    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb" event={"ID":"b491aad8-785b-4e38-b7d3-d4bafdb77743","Type":"ContainerStarted","Data":"e5c8880fa577fca1c713fcde89b8173f855fc18f9f0fe46f97041dc5c0748a5d"}
Sep 17 19:11:58.808393 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:58.808270    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:58.908854 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:11:58.908808    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:01.688515 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:12:01.688416    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb" event={"ID":"b491aad8-785b-4e38-b7d3-d4bafdb77743","Type":"ContainerStarted","Data":"57c3c869348fded2049739335673b8e521b5de3727c7082ffe8b9e4297f99b0e"}
Sep 17 19:12:01.689062 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:12:01.688576    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"
Sep 17 19:12:01.709044 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:12:01.708984    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb" podStartSLOduration=2.318732886 podStartE2EDuration="4.708966205s" totalImagesPullingTime="2.390233319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:01.708021523 +0000 UTC m=+344.634239959" watchObservedRunningTime="2026-09-17 19:12:01.708966205 +0000 UTC m=+344.635184628"
Sep 17 19:12:08.854405 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:12:08.854344    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:32.694925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:12:32.694886    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-9bzlb"
Sep 17 19:13:03.798766 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:03.798728    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:03.860490 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:03.860449    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:14.539932 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:14.539891    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c6586659-vml5x"]
Sep 17 19:13:14.657945 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:14.657834    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:14.760352 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:14.760309    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:14.829057 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:14.828971    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:15.678615 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:15.678581    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:15.765521 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:15.765481    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:15.927416 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:15.927382    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:16.025754 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:16.025660    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:16.125527 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:16.125496    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:17.194800 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:17.194758    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:17.267395 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:17.267351    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:39.567649 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.567560    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74c6586659-vml5x" podUID="190002cb-b170-4498-86ab-3b37fec3cc8a" containerName="console" containerID="cri-o://65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056" gracePeriod=15
Sep 17 19:13:39.817853 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.817782    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:13:39.934467 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.934420    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-service-ca\") pod \"190002cb-b170-4498-86ab-3b37fec3cc8a\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") "
Sep 17 19:13:39.934467 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.934481    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-serving-cert\") pod \"190002cb-b170-4498-86ab-3b37fec3cc8a\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") "
Sep 17 19:13:39.934733 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.934500    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-trusted-ca-bundle\") pod \"190002cb-b170-4498-86ab-3b37fec3cc8a\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") "
Sep 17 19:13:39.934733 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.934569    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-oauth-config\") pod \"190002cb-b170-4498-86ab-3b37fec3cc8a\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") "
Sep 17 19:13:39.934733 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.934609    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-console-config\") pod \"190002cb-b170-4498-86ab-3b37fec3cc8a\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") "
Sep 17 19:13:39.934733 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.934624    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm6s7\" (UniqueName: \"kubernetes.io/projected/190002cb-b170-4498-86ab-3b37fec3cc8a-kube-api-access-tm6s7\") pod \"190002cb-b170-4498-86ab-3b37fec3cc8a\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") "
Sep 17 19:13:39.934733 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.934667    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-oauth-serving-cert\") pod \"190002cb-b170-4498-86ab-3b37fec3cc8a\" (UID: \"190002cb-b170-4498-86ab-3b37fec3cc8a\") "
Sep 17 19:13:39.935072 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.935045    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "190002cb-b170-4498-86ab-3b37fec3cc8a" (UID: "190002cb-b170-4498-86ab-3b37fec3cc8a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:39.935072 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.935060    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-console-config" (OuterVolumeSpecName: "console-config") pod "190002cb-b170-4498-86ab-3b37fec3cc8a" (UID: "190002cb-b170-4498-86ab-3b37fec3cc8a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:39.935224 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.935036    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-service-ca" (OuterVolumeSpecName: "service-ca") pod "190002cb-b170-4498-86ab-3b37fec3cc8a" (UID: "190002cb-b170-4498-86ab-3b37fec3cc8a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:39.935224 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.935124    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "190002cb-b170-4498-86ab-3b37fec3cc8a" (UID: "190002cb-b170-4498-86ab-3b37fec3cc8a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:13:39.936856 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.936826    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190002cb-b170-4498-86ab-3b37fec3cc8a-kube-api-access-tm6s7" (OuterVolumeSpecName: "kube-api-access-tm6s7") pod "190002cb-b170-4498-86ab-3b37fec3cc8a" (UID: "190002cb-b170-4498-86ab-3b37fec3cc8a"). InnerVolumeSpecName "kube-api-access-tm6s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:13:39.937222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.937201    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "190002cb-b170-4498-86ab-3b37fec3cc8a" (UID: "190002cb-b170-4498-86ab-3b37fec3cc8a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:39.937297 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:39.937228    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "190002cb-b170-4498-86ab-3b37fec3cc8a" (UID: "190002cb-b170-4498-86ab-3b37fec3cc8a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:13:40.035400 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.035356    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-oauth-serving-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:13:40.035400 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.035394    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-service-ca\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:13:40.035400 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.035405    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-serving-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:13:40.035651 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.035414    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-trusted-ca-bundle\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:13:40.035651 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.035425    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/190002cb-b170-4498-86ab-3b37fec3cc8a-console-oauth-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:13:40.035651 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.035434    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/190002cb-b170-4498-86ab-3b37fec3cc8a-console-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:13:40.035651 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.035446    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm6s7\" (UniqueName: \"kubernetes.io/projected/190002cb-b170-4498-86ab-3b37fec3cc8a-kube-api-access-tm6s7\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:13:40.085762 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.085667    2567 generic.go:356] "Generic (PLEG): container finished" podID="190002cb-b170-4498-86ab-3b37fec3cc8a" containerID="65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056" exitCode=0
Sep 17 19:13:40.085762 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.085711    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c6586659-vml5x"
Sep 17 19:13:40.085762 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.085748    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c6586659-vml5x" event={"ID":"190002cb-b170-4498-86ab-3b37fec3cc8a","Type":"ContainerDied","Data":"65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056"}
Sep 17 19:13:40.086002 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.085805    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c6586659-vml5x" event={"ID":"190002cb-b170-4498-86ab-3b37fec3cc8a","Type":"ContainerDied","Data":"e51a14dae08e798e80b02807fd440e68da137c27a42d8dbbb72d3696a57ffed3"}
Sep 17 19:13:40.086002 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.085828    2567 scope.go:117] "RemoveContainer" containerID="65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056"
Sep 17 19:13:40.095491 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.095473    2567 scope.go:117] "RemoveContainer" containerID="65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056"
Sep 17 19:13:40.095850 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:13:40.095828    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056\": container with ID starting with 65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056 not found: ID does not exist" containerID="65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056"
Sep 17 19:13:40.095940 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.095861    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056"} err="failed to get container status \"65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056\": rpc error: code = NotFound desc = could not find container \"65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056\": container with ID starting with 65fb5a006eee5549efc5f0ed2d0e36c9bb0bdb6f005e31f9a4708d7bcdd62056 not found: ID does not exist"
Sep 17 19:13:40.118999 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.118963    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c6586659-vml5x"]
Sep 17 19:13:40.123878 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:40.123852    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74c6586659-vml5x"]
Sep 17 19:13:41.082384 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:41.082355    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="190002cb-b170-4498-86ab-3b37fec3cc8a" path="/var/lib/kubelet/pods/190002cb-b170-4498-86ab-3b37fec3cc8a/volumes"
Sep 17 19:13:41.461895 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:41.461854    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:13:41.548296 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:13:41.548262    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:51.851785 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:14:51.851733    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:51.910436 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:14:51.910404    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:59.649563 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:14:59.649525    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:05.246469 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.246375    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:05.464347 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.464304    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:05.562771 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.562682    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"]
Sep 17 19:15:05.563051 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.563035    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190002cb-b170-4498-86ab-3b37fec3cc8a" containerName="console"
Sep 17 19:15:05.563051 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.563053    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="190002cb-b170-4498-86ab-3b37fec3cc8a" containerName="console"
Sep 17 19:15:05.563215 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.563124    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="190002cb-b170-4498-86ab-3b37fec3cc8a" containerName="console"
Sep 17 19:15:05.563883 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.563854    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:05.566362 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.566344    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.571407 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.571377    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 19:15:05.571528 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.571514    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:15:05.571692 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.571677    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 19:15:05.572330 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.572301    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:15:05.572441 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.572390    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:15:05.572441 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.572426    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpx7x\""
Sep 17 19:15:05.572550 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.572462    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:15:05.580168 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.580129    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"]
Sep 17 19:15:05.635425 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.635381    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4nmz\" (UniqueName: \"kubernetes.io/projected/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-api-access-l4nmz\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.635649 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.635445    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c128fead-2a07-45ed-9ac1-417fb3860bd3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.635649 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.635503    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c128fead-2a07-45ed-9ac1-417fb3860bd3-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.635649 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.635540    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.635649 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.635584    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.692132 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.692070    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:05.736837 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.736794    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4nmz\" (UniqueName: \"kubernetes.io/projected/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-api-access-l4nmz\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.737049 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.736860    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c128fead-2a07-45ed-9ac1-417fb3860bd3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.737049 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.736886    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c128fead-2a07-45ed-9ac1-417fb3860bd3-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.737049 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.736905    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.737049 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.736936    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.737397 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:15:05.737116    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 19:15:05.737397 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:15:05.737215    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c128fead-2a07-45ed-9ac1-417fb3860bd3-proxy-tls podName:c128fead-2a07-45ed-9ac1-417fb3860bd3 nodeName:}" failed. No retries permitted until 2026-09-17 19:15:06.237191946 +0000 UTC m=+529.163410370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c128fead-2a07-45ed-9ac1-417fb3860bd3-proxy-tls") pod "isvc-canary-rollback-predictor-97977b6b5-t5fl8" (UID: "c128fead-2a07-45ed-9ac1-417fb3860bd3") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 19:15:05.737397 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.737364    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c128fead-2a07-45ed-9ac1-417fb3860bd3-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.737776 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.737750    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.737854 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.737754    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.745830 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.745803    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4nmz\" (UniqueName: \"kubernetes.io/projected/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-api-access-l4nmz\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:05.920790 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:05.920753    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:06.055418 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:06.055380    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:06.241123 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:06.241024    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c128fead-2a07-45ed-9ac1-417fb3860bd3-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:06.243693 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:06.243670    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c128fead-2a07-45ed-9ac1-417fb3860bd3-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-t5fl8\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:06.477326 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:06.477281    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:06.760086 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:06.760053    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"]
Sep 17 19:15:06.762970 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:15:06.762931    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc128fead_2a07_45ed_9ac1_417fb3860bd3.slice/crio-748535e42688acd14694a92896e7347b000862687f4af972183e737ae60ca87a WatchSource:0}: Error finding container 748535e42688acd14694a92896e7347b000862687f4af972183e737ae60ca87a: Status 404 returned error can't find the container with id 748535e42688acd14694a92896e7347b000862687f4af972183e737ae60ca87a
Sep 17 19:15:07.453254 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:07.453216    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" event={"ID":"c128fead-2a07-45ed-9ac1-417fb3860bd3","Type":"ContainerStarted","Data":"748535e42688acd14694a92896e7347b000862687f4af972183e737ae60ca87a"}
Sep 17 19:15:10.468463 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:10.468417    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" event={"ID":"c128fead-2a07-45ed-9ac1-417fb3860bd3","Type":"ContainerStarted","Data":"f02c97d79ce87b3af4fd5688213aa3358bd80c743ceb35ac3bbf90e691b92071"}
Sep 17 19:15:14.487794 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:14.487753    2567 generic.go:356] "Generic (PLEG): container finished" podID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerID="f02c97d79ce87b3af4fd5688213aa3358bd80c743ceb35ac3bbf90e691b92071" exitCode=0
Sep 17 19:15:14.488232 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:14.487825    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" event={"ID":"c128fead-2a07-45ed-9ac1-417fb3860bd3","Type":"ContainerDied","Data":"f02c97d79ce87b3af4fd5688213aa3358bd80c743ceb35ac3bbf90e691b92071"}
Sep 17 19:15:28.587678 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:28.587637    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" event={"ID":"c128fead-2a07-45ed-9ac1-417fb3860bd3","Type":"ContainerStarted","Data":"ef5e8e9db36d21dbf94ecb48e3e2f10ee0623c66fabf8ed7ef207ed411a1414c"}
Sep 17 19:15:30.599183 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:30.599076    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" event={"ID":"c128fead-2a07-45ed-9ac1-417fb3860bd3","Type":"ContainerStarted","Data":"313f78d15a141ae7c7093d8687782096c815e8fab558cfcc4c510778852d8a9a"}
Sep 17 19:15:30.599558 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:30.599295    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:30.627678 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:30.627606    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podStartSLOduration=2.974304586 podStartE2EDuration="25.627586856s" totalImagesPullingTime="18.729198085s" totalInitContainerRuntime="3.924084185s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:15:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:15:30.623899722 +0000 UTC m=+553.550118168" watchObservedRunningTime="2026-09-17 19:15:30.627586856 +0000 UTC m=+553.553805279"
Sep 17 19:15:31.604512 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:31.604481    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:31.605968 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:31.605930    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 19:15:32.608823 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:32.608778    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 19:15:37.613232 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:37.613200    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:15:37.613853 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:37.613825    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 19:15:47.614197 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:47.614155    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 19:15:57.613811 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:15:57.613769    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 19:16:07.614489 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:07.614442    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 19:16:17.613788 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:17.613743    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 19:16:19.010986 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:19.010953    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:16:19.013758 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:19.013734    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:16:19.015614 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:19.015593    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:16:19.018309 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:19.018287    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:16:27.614509 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:27.614453    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 19:16:37.613776 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:37.613688    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 19:16:45.837947 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:45.837905    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"]
Sep 17 19:16:45.838493 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:45.838240    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container" containerID="cri-o://ef5e8e9db36d21dbf94ecb48e3e2f10ee0623c66fabf8ed7ef207ed411a1414c" gracePeriod=30
Sep 17 19:16:45.838493 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:45.838274    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kube-rbac-proxy" containerID="cri-o://313f78d15a141ae7c7093d8687782096c815e8fab558cfcc4c510778852d8a9a" gracePeriod=30
Sep 17 19:16:45.838493 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:45.838454    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 17 19:16:46.932607 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:46.932569    2567 generic.go:356] "Generic (PLEG): container finished" podID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerID="313f78d15a141ae7c7093d8687782096c815e8fab558cfcc4c510778852d8a9a" exitCode=2
Sep 17 19:16:46.932996 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:46.932642    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" event={"ID":"c128fead-2a07-45ed-9ac1-417fb3860bd3","Type":"ContainerDied","Data":"313f78d15a141ae7c7093d8687782096c815e8fab558cfcc4c510778852d8a9a"}
Sep 17 19:16:47.609252 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:47.609214    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 19:16:52.609449 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:52.609396    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 19:16:55.319475 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:55.319442    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:55.840637 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:55.840604    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:16:57.609766 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:57.609712    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 19:16:57.610235 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:16:57.609851    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:17:00.764630 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:00.764597    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:00.877077 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:00.877043    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:01.010948 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:01.010915    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:01.144088 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:01.144057    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:01.338490 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:01.338450    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:02.609480 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:02.609429    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 19:17:07.610012 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:07.609965    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 19:17:12.609697 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:12.609656    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 19:17:16.054392 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.054357    2567 generic.go:356] "Generic (PLEG): container finished" podID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerID="ef5e8e9db36d21dbf94ecb48e3e2f10ee0623c66fabf8ed7ef207ed411a1414c" exitCode=137
Sep 17 19:17:16.054793 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.054428    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" event={"ID":"c128fead-2a07-45ed-9ac1-417fb3860bd3","Type":"ContainerDied","Data":"ef5e8e9db36d21dbf94ecb48e3e2f10ee0623c66fabf8ed7ef207ed411a1414c"}
Sep 17 19:17:16.490552 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.490526    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:17:16.625126 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.625091    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c128fead-2a07-45ed-9ac1-417fb3860bd3-proxy-tls\") pod \"c128fead-2a07-45ed-9ac1-417fb3860bd3\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") "
Sep 17 19:17:16.625340 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.625195    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-cabundle-cert\") pod \"c128fead-2a07-45ed-9ac1-417fb3860bd3\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") "
Sep 17 19:17:16.625340 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.625234    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4nmz\" (UniqueName: \"kubernetes.io/projected/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-api-access-l4nmz\") pod \"c128fead-2a07-45ed-9ac1-417fb3860bd3\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") "
Sep 17 19:17:16.625340 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.625265    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c128fead-2a07-45ed-9ac1-417fb3860bd3-kserve-provision-location\") pod \"c128fead-2a07-45ed-9ac1-417fb3860bd3\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") "
Sep 17 19:17:16.625340 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.625322    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-rbac-proxy-sar-config\") pod \"c128fead-2a07-45ed-9ac1-417fb3860bd3\" (UID: \"c128fead-2a07-45ed-9ac1-417fb3860bd3\") "
Sep 17 19:17:16.625697 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.625667    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c128fead-2a07-45ed-9ac1-417fb3860bd3" (UID: "c128fead-2a07-45ed-9ac1-417fb3860bd3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:17:16.625804 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.625684    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c128fead-2a07-45ed-9ac1-417fb3860bd3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c128fead-2a07-45ed-9ac1-417fb3860bd3" (UID: "c128fead-2a07-45ed-9ac1-417fb3860bd3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:17:16.625804 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.625720    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c128fead-2a07-45ed-9ac1-417fb3860bd3" (UID: "c128fead-2a07-45ed-9ac1-417fb3860bd3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:17:16.627577 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.627541    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c128fead-2a07-45ed-9ac1-417fb3860bd3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c128fead-2a07-45ed-9ac1-417fb3860bd3" (UID: "c128fead-2a07-45ed-9ac1-417fb3860bd3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:17:16.627682 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.627571    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-api-access-l4nmz" (OuterVolumeSpecName: "kube-api-access-l4nmz") pod "c128fead-2a07-45ed-9ac1-417fb3860bd3" (UID: "c128fead-2a07-45ed-9ac1-417fb3860bd3"). InnerVolumeSpecName "kube-api-access-l4nmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:17:16.726913 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.726859    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:17:16.726913 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.726905    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c128fead-2a07-45ed-9ac1-417fb3860bd3-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:17:16.726913 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.726916    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c128fead-2a07-45ed-9ac1-417fb3860bd3-cabundle-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:17:16.726913 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.726925    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4nmz\" (UniqueName: \"kubernetes.io/projected/c128fead-2a07-45ed-9ac1-417fb3860bd3-kube-api-access-l4nmz\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:17:16.726913 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:16.726934    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c128fead-2a07-45ed-9ac1-417fb3860bd3-kserve-provision-location\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:17:17.061183 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:17.061083    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8" event={"ID":"c128fead-2a07-45ed-9ac1-417fb3860bd3","Type":"ContainerDied","Data":"748535e42688acd14694a92896e7347b000862687f4af972183e737ae60ca87a"}
Sep 17 19:17:17.061183 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:17.061172    2567 scope.go:117] "RemoveContainer" containerID="313f78d15a141ae7c7093d8687782096c815e8fab558cfcc4c510778852d8a9a"
Sep 17 19:17:17.061183 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:17.061094    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"
Sep 17 19:17:17.070215 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:17.070194    2567 scope.go:117] "RemoveContainer" containerID="ef5e8e9db36d21dbf94ecb48e3e2f10ee0623c66fabf8ed7ef207ed411a1414c"
Sep 17 19:17:17.078161 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:17.078126    2567 scope.go:117] "RemoveContainer" containerID="f02c97d79ce87b3af4fd5688213aa3358bd80c743ceb35ac3bbf90e691b92071"
Sep 17 19:17:17.085715 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:17.085686    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"]
Sep 17 19:17:17.090760 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:17.090734    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-t5fl8"]
Sep 17 19:17:19.083460 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:17:19.083424    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" path="/var/lib/kubelet/pods/c128fead-2a07-45ed-9ac1-417fb3860bd3/volumes"
Sep 17 19:18:16.397102 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:16.397010    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:16.498783 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:16.498748    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:25.340839 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:25.340805    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:30.479002 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.478959    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:30.523172 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.523121    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"]
Sep 17 19:18:30.523475 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.523461    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kube-rbac-proxy"
Sep 17 19:18:30.523519 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.523476    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kube-rbac-proxy"
Sep 17 19:18:30.523519 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.523493    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container"
Sep 17 19:18:30.523519 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.523499    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container"
Sep 17 19:18:30.523519 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.523506    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="storage-initializer"
Sep 17 19:18:30.523519 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.523512    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="storage-initializer"
Sep 17 19:18:30.523669 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.523571    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kserve-container"
Sep 17 19:18:30.523669 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.523578    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c128fead-2a07-45ed-9ac1-417fb3860bd3" containerName="kube-rbac-proxy"
Sep 17 19:18:30.526767 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.526746    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.529430 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.529359    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:18:30.529596 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.529542    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:18:30.529596 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.529542    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 19:18:30.529699 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.529624    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:18:30.529699 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.529625    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 19:18:30.529840 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.529823    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:18:30.530679 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.530664    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpx7x\""
Sep 17 19:18:30.538002 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.537975    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"]
Sep 17 19:18:30.582211 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.582169    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:30.652093 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.652058    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhd2k\" (UniqueName: \"kubernetes.io/projected/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-api-access-fhd2k\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.652332 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.652108    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.652332 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.652172    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00473f57-9db4-4b5f-a323-edf541cc81a7-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.652332 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.652225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.652332 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.652294    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00473f57-9db4-4b5f-a323-edf541cc81a7-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.665514 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.665477    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:30.748533 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.748452    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:30.753574 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.753547    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00473f57-9db4-4b5f-a323-edf541cc81a7-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.753647 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.753593    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhd2k\" (UniqueName: \"kubernetes.io/projected/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-api-access-fhd2k\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.753716 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.753702    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.753750 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.753733    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00473f57-9db4-4b5f-a323-edf541cc81a7-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.753787 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.753762    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.754005 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.753973    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00473f57-9db4-4b5f-a323-edf541cc81a7-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.754556 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.754524    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.754652 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.754524    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.756345 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.756317    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00473f57-9db4-4b5f-a323-edf541cc81a7-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.765662 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.765638    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhd2k\" (UniqueName: \"kubernetes.io/projected/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-api-access-fhd2k\") pod \"isvc-canary-stop-v2-predictor-d65d59944-5dj7q\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:30.837719 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:30.837677    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:31.115234 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:31.115133    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"]
Sep 17 19:18:31.117579 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:18:31.117550    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00473f57_9db4_4b5f_a323_edf541cc81a7.slice/crio-2fef5f6ac9538d25d492251cbe9c5b883e6b7df80a9f96f3cd149da19d7732b3 WatchSource:0}: Error finding container 2fef5f6ac9538d25d492251cbe9c5b883e6b7df80a9f96f3cd149da19d7732b3: Status 404 returned error can't find the container with id 2fef5f6ac9538d25d492251cbe9c5b883e6b7df80a9f96f3cd149da19d7732b3
Sep 17 19:18:31.362773 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:31.362732    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" event={"ID":"00473f57-9db4-4b5f-a323-edf541cc81a7","Type":"ContainerStarted","Data":"01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9"}
Sep 17 19:18:31.363063 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:31.362801    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" event={"ID":"00473f57-9db4-4b5f-a323-edf541cc81a7","Type":"ContainerStarted","Data":"2fef5f6ac9538d25d492251cbe9c5b883e6b7df80a9f96f3cd149da19d7732b3"}
Sep 17 19:18:35.379882 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:35.379840    2567 generic.go:356] "Generic (PLEG): container finished" podID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerID="01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9" exitCode=0
Sep 17 19:18:35.380313 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:35.379883    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" event={"ID":"00473f57-9db4-4b5f-a323-edf541cc81a7","Type":"ContainerDied","Data":"01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9"}
Sep 17 19:18:36.386817 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:36.386780    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" event={"ID":"00473f57-9db4-4b5f-a323-edf541cc81a7","Type":"ContainerStarted","Data":"1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732"}
Sep 17 19:18:36.387288 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:36.386843    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" event={"ID":"00473f57-9db4-4b5f-a323-edf541cc81a7","Type":"ContainerStarted","Data":"11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569"}
Sep 17 19:18:36.387288 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:36.387129    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:36.387368 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:36.387302    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:36.388701 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:36.388676    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 19:18:36.409016 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:36.408964    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podStartSLOduration=2.744887948 podStartE2EDuration="6.408943457s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.664055509s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:18:36.408020762 +0000 UTC m=+739.334239195" watchObservedRunningTime="2026-09-17 19:18:36.408943457 +0000 UTC m=+739.335161879"
Sep 17 19:18:37.391007 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:37.390967    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 19:18:40.872755 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:40.872716    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:41.433557 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:41.433517    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:42.395649 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:42.395617    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:18:42.396121 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:42.396094    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 19:18:50.966683 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:50.966643    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:51.078205 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.078164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:51.171489 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.171457    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:51.271863 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.271787    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:51.319753 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.319711    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"]
Sep 17 19:18:51.323414 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.323395    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.325890 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.325861    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-7eb14-predictor-serving-cert\""
Sep 17 19:18:51.326010 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.325899    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-7eb14-kube-rbac-proxy-sar-config\""
Sep 17 19:18:51.333001 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.332972    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"]
Sep 17 19:18:51.367270 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.367230    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:51.433201 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.433164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:51.437387 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.437362    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s659d\" (UniqueName: \"kubernetes.io/projected/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-api-access-s659d\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.437488 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.437441    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca4a762-ef24-4015-ab6e-fcac25afcb45-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.437488 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.437462    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca4a762-ef24-4015-ab6e-fcac25afcb45-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.437581 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.437497    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.437581 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.437516    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.538573 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.538470    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.538573 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.538519    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.538573 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.538553    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s659d\" (UniqueName: \"kubernetes.io/projected/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-api-access-s659d\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.538867 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.538636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca4a762-ef24-4015-ab6e-fcac25afcb45-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.538867 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.538660    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca4a762-ef24-4015-ab6e-fcac25afcb45-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.538867 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:18:51.538790    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-7eb14-predictor-serving-cert" not found
Sep 17 19:18:51.539024 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:18:51.538887    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca4a762-ef24-4015-ab6e-fcac25afcb45-proxy-tls podName:bca4a762-ef24-4015-ab6e-fcac25afcb45 nodeName:}" failed. No retries permitted until 2026-09-17 19:18:52.038862221 +0000 UTC m=+754.965080639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bca4a762-ef24-4015-ab6e-fcac25afcb45-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" (UID: "bca4a762-ef24-4015-ab6e-fcac25afcb45") : secret "isvc-xgboost-graph-raw-hpa-7eb14-predictor-serving-cert" not found
Sep 17 19:18:51.539092 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.539045    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca4a762-ef24-4015-ab6e-fcac25afcb45-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.539255 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.539237    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.539299 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.539271    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.551079 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.550823    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s659d\" (UniqueName: \"kubernetes.io/projected/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-api-access-s659d\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:51.598883 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:51.598843    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:18:52.043559 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:52.043522    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca4a762-ef24-4015-ab6e-fcac25afcb45-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:52.046060 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:52.046036    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca4a762-ef24-4015-ab6e-fcac25afcb45-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:52.235061 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:52.235017    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:18:52.396634 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:52.396594    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 19:18:52.516057 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:52.516029    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"]
Sep 17 19:18:52.520649 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:18:52.520609    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca4a762_ef24_4015_ab6e_fcac25afcb45.slice/crio-0b87c05decca266aa8e06ce4288a4b4d1ddb7646360eac76bd40ab89303fda74 WatchSource:0}: Error finding container 0b87c05decca266aa8e06ce4288a4b4d1ddb7646360eac76bd40ab89303fda74: Status 404 returned error can't find the container with id 0b87c05decca266aa8e06ce4288a4b4d1ddb7646360eac76bd40ab89303fda74
Sep 17 19:18:53.457563 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:53.457527    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" event={"ID":"bca4a762-ef24-4015-ab6e-fcac25afcb45","Type":"ContainerStarted","Data":"582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771"}
Sep 17 19:18:53.457959 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:53.457588    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" event={"ID":"bca4a762-ef24-4015-ab6e-fcac25afcb45","Type":"ContainerStarted","Data":"0b87c05decca266aa8e06ce4288a4b4d1ddb7646360eac76bd40ab89303fda74"}
Sep 17 19:18:56.471515 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:56.471469    2567 generic.go:356] "Generic (PLEG): container finished" podID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerID="582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771" exitCode=0
Sep 17 19:18:56.471946 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:56.471537    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" event={"ID":"bca4a762-ef24-4015-ab6e-fcac25afcb45","Type":"ContainerDied","Data":"582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771"}
Sep 17 19:18:56.472955 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:56.472932    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:18:56.972759 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:18:56.972678    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:02.396984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:02.396943    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 19:19:12.396519 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:12.396475    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 19:19:18.583262 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:18.583220    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" event={"ID":"bca4a762-ef24-4015-ab6e-fcac25afcb45","Type":"ContainerStarted","Data":"af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033"}
Sep 17 19:19:18.583706 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:18.583289    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" event={"ID":"bca4a762-ef24-4015-ab6e-fcac25afcb45","Type":"ContainerStarted","Data":"861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6"}
Sep 17 19:19:18.583706 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:18.583568    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:19:18.606095 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:18.606040    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podStartSLOduration=2.317679341 podStartE2EDuration="27.606020975s" totalImagesPullingTime="21.463302756s" totalInitContainerRuntime="3.825038878s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:19:18.604067779 +0000 UTC m=+781.530286213" watchObservedRunningTime="2026-09-17 19:19:18.606020975 +0000 UTC m=+781.532239397"
Sep 17 19:19:19.588041 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:19.588006    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:19:19.589539 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:19.589503    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:19:20.592663 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:20.592622    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:19:22.396218 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:22.396180    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 19:19:25.597400 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:25.597367    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:19:25.597925 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:25.597899    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:19:32.396418 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:32.396335    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 19:19:35.598931 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:35.598888    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:19:42.396403 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:42.396353    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 19:19:45.598204 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:45.598110    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:19:52.397120 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:52.397081    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 19:19:55.598323 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:19:55.598280    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:20:00.078485 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:00.078429    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 19:20:05.598778 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:05.598736    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:20:10.079156 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:10.079115    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:20:10.597688 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:10.597652    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"]
Sep 17 19:20:10.603302 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:10.603264    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:10.684903 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:10.684871    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:10.805976 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:10.805934    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container" containerID="cri-o://11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569" gracePeriod=30
Sep 17 19:20:10.805976 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:10.805963    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kube-rbac-proxy" containerID="cri-o://1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732" gracePeriod=30
Sep 17 19:20:11.812710 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:11.812674    2567 generic.go:356] "Generic (PLEG): container finished" podID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerID="1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732" exitCode=2
Sep 17 19:20:11.813093 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:11.812737    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" event={"ID":"00473f57-9db4-4b5f-a323-edf541cc81a7","Type":"ContainerDied","Data":"1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732"}
Sep 17 19:20:12.391615 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:12.391568    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 17 19:20:15.598812 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:15.598764    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:20:16.457936 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.457908    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:20:16.607223 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.607106    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhd2k\" (UniqueName: \"kubernetes.io/projected/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-api-access-fhd2k\") pod \"00473f57-9db4-4b5f-a323-edf541cc81a7\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") "
Sep 17 19:20:16.607223 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.607173    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-cabundle-cert\") pod \"00473f57-9db4-4b5f-a323-edf541cc81a7\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") "
Sep 17 19:20:16.607223 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.607217    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00473f57-9db4-4b5f-a323-edf541cc81a7-kserve-provision-location\") pod \"00473f57-9db4-4b5f-a323-edf541cc81a7\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") "
Sep 17 19:20:16.607788 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.607240    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00473f57-9db4-4b5f-a323-edf541cc81a7-proxy-tls\") pod \"00473f57-9db4-4b5f-a323-edf541cc81a7\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") "
Sep 17 19:20:16.607788 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.607257    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-rbac-proxy-sar-config\") pod \"00473f57-9db4-4b5f-a323-edf541cc81a7\" (UID: \"00473f57-9db4-4b5f-a323-edf541cc81a7\") "
Sep 17 19:20:16.607788 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.607604    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00473f57-9db4-4b5f-a323-edf541cc81a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "00473f57-9db4-4b5f-a323-edf541cc81a7" (UID: "00473f57-9db4-4b5f-a323-edf541cc81a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:20:16.607788 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.607631    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "00473f57-9db4-4b5f-a323-edf541cc81a7" (UID: "00473f57-9db4-4b5f-a323-edf541cc81a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:16.607788 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.607656    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "00473f57-9db4-4b5f-a323-edf541cc81a7" (UID: "00473f57-9db4-4b5f-a323-edf541cc81a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:16.609470 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.609447    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00473f57-9db4-4b5f-a323-edf541cc81a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "00473f57-9db4-4b5f-a323-edf541cc81a7" (UID: "00473f57-9db4-4b5f-a323-edf541cc81a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:20:16.609527 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.609467    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-api-access-fhd2k" (OuterVolumeSpecName: "kube-api-access-fhd2k") pod "00473f57-9db4-4b5f-a323-edf541cc81a7" (UID: "00473f57-9db4-4b5f-a323-edf541cc81a7"). InnerVolumeSpecName "kube-api-access-fhd2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:20:16.708654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.708595    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhd2k\" (UniqueName: \"kubernetes.io/projected/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-api-access-fhd2k\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:20:16.708654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.708642    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-cabundle-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:20:16.708654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.708653    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/00473f57-9db4-4b5f-a323-edf541cc81a7-kserve-provision-location\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:20:16.708654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.708662    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/00473f57-9db4-4b5f-a323-edf541cc81a7-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:20:16.708654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.708671    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/00473f57-9db4-4b5f-a323-edf541cc81a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:20:16.836774 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.836736    2567 generic.go:356] "Generic (PLEG): container finished" podID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerID="11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569" exitCode=0
Sep 17 19:20:16.836965 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.836795    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"
Sep 17 19:20:16.836965 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.836797    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" event={"ID":"00473f57-9db4-4b5f-a323-edf541cc81a7","Type":"ContainerDied","Data":"11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569"}
Sep 17 19:20:16.836965 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.836949    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q" event={"ID":"00473f57-9db4-4b5f-a323-edf541cc81a7","Type":"ContainerDied","Data":"2fef5f6ac9538d25d492251cbe9c5b883e6b7df80a9f96f3cd149da19d7732b3"}
Sep 17 19:20:16.837123 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.837000    2567 scope.go:117] "RemoveContainer" containerID="1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732"
Sep 17 19:20:16.846047 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.846030    2567 scope.go:117] "RemoveContainer" containerID="11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569"
Sep 17 19:20:16.854106 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.854087    2567 scope.go:117] "RemoveContainer" containerID="01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9"
Sep 17 19:20:16.860429 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.860402    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"]
Sep 17 19:20:16.862962 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.862939    2567 scope.go:117] "RemoveContainer" containerID="1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732"
Sep 17 19:20:16.863312 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:20:16.863293    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732\": container with ID starting with 1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732 not found: ID does not exist" containerID="1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732"
Sep 17 19:20:16.863372 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.863322    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732"} err="failed to get container status \"1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732\": rpc error: code = NotFound desc = could not find container \"1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732\": container with ID starting with 1ac606f66917315c72808e756182b9d83729e36264dd7f3ac22c757ba1bd9732 not found: ID does not exist"
Sep 17 19:20:16.863372 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.863341    2567 scope.go:117] "RemoveContainer" containerID="11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569"
Sep 17 19:20:16.863662 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:20:16.863643    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569\": container with ID starting with 11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569 not found: ID does not exist" containerID="11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569"
Sep 17 19:20:16.863717 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.863669    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569"} err="failed to get container status \"11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569\": rpc error: code = NotFound desc = could not find container \"11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569\": container with ID starting with 11c9cfe322c80ecb7d8b0c632c1db8de113663d08627d3114750de35a6fdb569 not found: ID does not exist"
Sep 17 19:20:16.863717 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.863688    2567 scope.go:117] "RemoveContainer" containerID="01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9"
Sep 17 19:20:16.863942 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:20:16.863915    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9\": container with ID starting with 01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9 not found: ID does not exist" containerID="01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9"
Sep 17 19:20:16.863997 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.863952    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9"} err="failed to get container status \"01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9\": rpc error: code = NotFound desc = could not find container \"01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9\": container with ID starting with 01af1c08e8b59c2393003253e2ffcae19a6088d776d2761df9c17772a1c633c9 not found: ID does not exist"
Sep 17 19:20:16.866811 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:16.866784    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-5dj7q"]
Sep 17 19:20:17.083051 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:17.083017    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" path="/var/lib/kubelet/pods/00473f57-9db4-4b5f-a323-edf541cc81a7/volumes"
Sep 17 19:20:20.802375 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:20.802337    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:21.343964 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:21.343929    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:25.598691 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:25.598649    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 19:20:29.084130 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:29.084093    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:20:31.195701 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:31.195666    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:31.267039 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:31.267003    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:33.355489 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:33.355451    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:41.263679 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.263615    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:41.370817 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.370775    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:41.381323 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.381292    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"]
Sep 17 19:20:41.381573 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.381541    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container" containerID="cri-o://861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6" gracePeriod=30
Sep 17 19:20:41.381693 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.381568    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kube-rbac-proxy" containerID="cri-o://af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033" gracePeriod=30
Sep 17 19:20:41.469567 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.469522    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:41.470166 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.470118    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"]
Sep 17 19:20:41.470520 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.470503    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="storage-initializer"
Sep 17 19:20:41.470592 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.470523    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="storage-initializer"
Sep 17 19:20:41.470592 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.470536    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kube-rbac-proxy"
Sep 17 19:20:41.470592 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.470544    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kube-rbac-proxy"
Sep 17 19:20:41.470592 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.470563    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container"
Sep 17 19:20:41.470592 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.470573    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container"
Sep 17 19:20:41.470893 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.470680    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kserve-container"
Sep 17 19:20:41.470893 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.470694    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="00473f57-9db4-4b5f-a323-edf541cc81a7" containerName="kube-rbac-proxy"
Sep 17 19:20:41.477305 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.476186    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:41.480575 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.479898    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-09f87-kube-rbac-proxy-sar-config\""
Sep 17 19:20:41.480575 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.480179    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-09f87-predictor-serving-cert\""
Sep 17 19:20:41.482047 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.482022    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"]
Sep 17 19:20:41.516812 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.516716    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0ec727d-e510-410c-bb6a-24a73b423c7c-proxy-tls\") pod \"message-dumper-raw-09f87-predictor-6547b66f58-bhnwg\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") " pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:41.516812 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.516805    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-09f87-predictor-6547b66f58-bhnwg\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") " pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:41.517011 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.516831    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwk2b\" (UniqueName: \"kubernetes.io/projected/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-api-access-gwk2b\") pod \"message-dumper-raw-09f87-predictor-6547b66f58-bhnwg\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") " pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:41.528068 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.528038    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:41.617401 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.617363    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-09f87-predictor-6547b66f58-bhnwg\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") " pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:41.617591 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.617416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwk2b\" (UniqueName: \"kubernetes.io/projected/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-api-access-gwk2b\") pod \"message-dumper-raw-09f87-predictor-6547b66f58-bhnwg\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") " pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:41.617591 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.617458    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0ec727d-e510-410c-bb6a-24a73b423c7c-proxy-tls\") pod \"message-dumper-raw-09f87-predictor-6547b66f58-bhnwg\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") " pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:41.618126 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.618107    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-09f87-predictor-6547b66f58-bhnwg\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") " pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:41.619998 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.619978    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0ec727d-e510-410c-bb6a-24a73b423c7c-proxy-tls\") pod \"message-dumper-raw-09f87-predictor-6547b66f58-bhnwg\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") " pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:41.626179 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.626118    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwk2b\" (UniqueName: \"kubernetes.io/projected/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-api-access-gwk2b\") pod \"message-dumper-raw-09f87-predictor-6547b66f58-bhnwg\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") " pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:41.650294 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.650258    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:41.788971 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.788858    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:41.942522 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.942486    2567 generic.go:356] "Generic (PLEG): container finished" podID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerID="af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033" exitCode=2
Sep 17 19:20:41.942695 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:41.942557    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" event={"ID":"bca4a762-ef24-4015-ab6e-fcac25afcb45","Type":"ContainerDied","Data":"af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033"}
Sep 17 19:20:42.061501 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:42.061472    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"]
Sep 17 19:20:42.064009 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:20:42.063970    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ec727d_e510_410c_bb6a_24a73b423c7c.slice/crio-f85e4d38a4692cb8c6458c6f4d778a4b34e326ec7a0bfa28f4343f9d0cbe8ee1 WatchSource:0}: Error finding container f85e4d38a4692cb8c6458c6f4d778a4b34e326ec7a0bfa28f4343f9d0cbe8ee1: Status 404 returned error can't find the container with id f85e4d38a4692cb8c6458c6f4d778a4b34e326ec7a0bfa28f4343f9d0cbe8ee1
Sep 17 19:20:42.949206 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:42.949172    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg" event={"ID":"b0ec727d-e510-410c-bb6a-24a73b423c7c","Type":"ContainerStarted","Data":"f85e4d38a4692cb8c6458c6f4d778a4b34e326ec7a0bfa28f4343f9d0cbe8ee1"}
Sep 17 19:20:43.955773 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:43.955740    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg" event={"ID":"b0ec727d-e510-410c-bb6a-24a73b423c7c","Type":"ContainerStarted","Data":"dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf"}
Sep 17 19:20:43.956245 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:43.955801    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg" event={"ID":"b0ec727d-e510-410c-bb6a-24a73b423c7c","Type":"ContainerStarted","Data":"f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5"}
Sep 17 19:20:43.956245 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:43.955837    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:43.956245 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:43.955975    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:43.957840 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:43.957815    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:43.976671 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:43.976622    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg" podStartSLOduration=1.716982537 podStartE2EDuration="2.97660186s" totalImagesPullingTime="1.259619323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:20:43.974522481 +0000 UTC m=+866.900740905" watchObservedRunningTime="2026-09-17 19:20:43.97660186 +0000 UTC m=+866.902820283"
Sep 17 19:20:45.593795 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.593746    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 19:20:45.942846 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.942820    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:20:45.954293 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.954266    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s659d\" (UniqueName: \"kubernetes.io/projected/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-api-access-s659d\") pod \"bca4a762-ef24-4015-ab6e-fcac25afcb45\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") "
Sep 17 19:20:45.954426 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.954330    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-cabundle-cert\") pod \"bca4a762-ef24-4015-ab6e-fcac25afcb45\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") "
Sep 17 19:20:45.954426 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.954366    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca4a762-ef24-4015-ab6e-fcac25afcb45-proxy-tls\") pod \"bca4a762-ef24-4015-ab6e-fcac25afcb45\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") "
Sep 17 19:20:45.954426 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.954416    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-rbac-proxy-sar-config\") pod \"bca4a762-ef24-4015-ab6e-fcac25afcb45\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") "
Sep 17 19:20:45.954602 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.954461    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca4a762-ef24-4015-ab6e-fcac25afcb45-kserve-provision-location\") pod \"bca4a762-ef24-4015-ab6e-fcac25afcb45\" (UID: \"bca4a762-ef24-4015-ab6e-fcac25afcb45\") "
Sep 17 19:20:45.954782 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.954757    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bca4a762-ef24-4015-ab6e-fcac25afcb45" (UID: "bca4a762-ef24-4015-ab6e-fcac25afcb45"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:45.954872 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.954773    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca4a762-ef24-4015-ab6e-fcac25afcb45-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bca4a762-ef24-4015-ab6e-fcac25afcb45" (UID: "bca4a762-ef24-4015-ab6e-fcac25afcb45"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:20:45.954872 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.954811    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bca4a762-ef24-4015-ab6e-fcac25afcb45" (UID: "bca4a762-ef24-4015-ab6e-fcac25afcb45"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:20:45.956580 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.956554    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca4a762-ef24-4015-ab6e-fcac25afcb45-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bca4a762-ef24-4015-ab6e-fcac25afcb45" (UID: "bca4a762-ef24-4015-ab6e-fcac25afcb45"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:20:45.956680 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.956556    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-api-access-s659d" (OuterVolumeSpecName: "kube-api-access-s659d") pod "bca4a762-ef24-4015-ab6e-fcac25afcb45" (UID: "bca4a762-ef24-4015-ab6e-fcac25afcb45"). InnerVolumeSpecName "kube-api-access-s659d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:20:45.967440 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.967405    2567 generic.go:356] "Generic (PLEG): container finished" podID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerID="861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6" exitCode=0
Sep 17 19:20:45.967928 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.967510    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" event={"ID":"bca4a762-ef24-4015-ab6e-fcac25afcb45","Type":"ContainerDied","Data":"861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6"}
Sep 17 19:20:45.967928 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.967625    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7" event={"ID":"bca4a762-ef24-4015-ab6e-fcac25afcb45","Type":"ContainerDied","Data":"0b87c05decca266aa8e06ce4288a4b4d1ddb7646360eac76bd40ab89303fda74"}
Sep 17 19:20:45.967928 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.967665    2567 scope.go:117] "RemoveContainer" containerID="af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033"
Sep 17 19:20:45.967928 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.967526    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"
Sep 17 19:20:45.977192 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.977019    2567 scope.go:117] "RemoveContainer" containerID="861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6"
Sep 17 19:20:45.986741 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.986722    2567 scope.go:117] "RemoveContainer" containerID="582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771"
Sep 17 19:20:45.993040 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.993009    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"]
Sep 17 19:20:45.996779 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.996759    2567 scope.go:117] "RemoveContainer" containerID="af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033"
Sep 17 19:20:45.997251 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:20:45.997226    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033\": container with ID starting with af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033 not found: ID does not exist" containerID="af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033"
Sep 17 19:20:45.997319 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.997250    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-7eb14-predictor-57f56f9dd9-rbkp7"]
Sep 17 19:20:45.997319 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.997262    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033"} err="failed to get container status \"af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033\": rpc error: code = NotFound desc = could not find container \"af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033\": container with ID starting with af2eeb4a92b102bf796db21912be0fff297c8b6b161b53d41da1479b81893033 not found: ID does not exist"
Sep 17 19:20:45.997319 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.997283    2567 scope.go:117] "RemoveContainer" containerID="861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6"
Sep 17 19:20:45.997585 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:20:45.997564    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6\": container with ID starting with 861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6 not found: ID does not exist" containerID="861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6"
Sep 17 19:20:45.997687 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.997595    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6"} err="failed to get container status \"861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6\": rpc error: code = NotFound desc = could not find container \"861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6\": container with ID starting with 861f2fc03a98ba510e186fc6f0e7d28bb938aae01fed108600296ba380fe29a6 not found: ID does not exist"
Sep 17 19:20:45.997687 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.997609    2567 scope.go:117] "RemoveContainer" containerID="582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771"
Sep 17 19:20:45.997902 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:20:45.997883    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771\": container with ID starting with 582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771 not found: ID does not exist" containerID="582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771"
Sep 17 19:20:45.997963 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:45.997911    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771"} err="failed to get container status \"582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771\": rpc error: code = NotFound desc = could not find container \"582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771\": container with ID starting with 582f0336efd1f208d1d5a92d5db5db9f58c5e78ca486835b90a0c9c29aa31771 not found: ID does not exist"
Sep 17 19:20:46.055371 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:46.055327    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bca4a762-ef24-4015-ab6e-fcac25afcb45-kserve-provision-location\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:20:46.055371 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:46.055363    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s659d\" (UniqueName: \"kubernetes.io/projected/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-api-access-s659d\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:20:46.055371 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:46.055380    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-cabundle-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:20:46.055683 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:46.055393    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bca4a762-ef24-4015-ab6e-fcac25afcb45-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:20:46.055683 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:46.055405    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bca4a762-ef24-4015-ab6e-fcac25afcb45-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:20:47.082788 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:47.082755    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" path="/var/lib/kubelet/pods/bca4a762-ef24-4015-ab6e-fcac25afcb45/volumes"
Sep 17 19:20:50.972752 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:50.972726    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:20:51.456317 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:51.456272    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:51.794403 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:20:51.794310    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:19.038054 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:21:19.038019    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:21:19.039207 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:21:19.039186    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:21:19.042543 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:21:19.042522    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:21:19.043582 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:21:19.043560    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:22:06.000183 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.000131    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:06.012842 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.012809    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"]
Sep 17 19:22:06.013156 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.013130    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="storage-initializer"
Sep 17 19:22:06.013206 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.013158    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="storage-initializer"
Sep 17 19:22:06.013206 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.013174    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container"
Sep 17 19:22:06.013206 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.013180    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container"
Sep 17 19:22:06.013206 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.013188    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kube-rbac-proxy"
Sep 17 19:22:06.013206 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.013193    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kube-rbac-proxy"
Sep 17 19:22:06.013384 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.013248    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kube-rbac-proxy"
Sep 17 19:22:06.013384 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.013255    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca4a762-ef24-4015-ab6e-fcac25afcb45" containerName="kserve-container"
Sep 17 19:22:06.016430 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.016411    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.020991 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.020955    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-5ad3e2\""
Sep 17 19:22:06.021183 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.021171    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-5ad3e2-dockercfg-f8l4g\""
Sep 17 19:22:06.021295 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.021201    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-5ad3e2-predictor-serving-cert\""
Sep 17 19:22:06.021473 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.021450    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-5ad3e2-kube-rbac-proxy-sar-config\""
Sep 17 19:22:06.022187 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.022168    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:22:06.028592 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.028569    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"]
Sep 17 19:22:06.149670 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.149628    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.149670 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.149677    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4216b47-a57d-465d-b826-7a485ba64ff2-proxy-tls\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.149890 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.149725    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-cabundle-cert\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.149890 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.149751    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldqd2\" (UniqueName: \"kubernetes.io/projected/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-api-access-ldqd2\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.149890 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.149795    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4216b47-a57d-465d-b826-7a485ba64ff2-kserve-provision-location\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.250523 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.250426    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.250523 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.250472    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4216b47-a57d-465d-b826-7a485ba64ff2-proxy-tls\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.250523 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.250510    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-cabundle-cert\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.250814 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.250540    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldqd2\" (UniqueName: \"kubernetes.io/projected/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-api-access-ldqd2\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.250814 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.250576    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4216b47-a57d-465d-b826-7a485ba64ff2-kserve-provision-location\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.251014 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.250988    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4216b47-a57d-465d-b826-7a485ba64ff2-kserve-provision-location\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.251222 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.251196    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.251271 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.251218    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-cabundle-cert\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.253059 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.253040    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4216b47-a57d-465d-b826-7a485ba64ff2-proxy-tls\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.260116 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.260091    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldqd2\" (UniqueName: \"kubernetes.io/projected/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-api-access-ldqd2\") pod \"isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") " pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.327091 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.327046    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:06.604496 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:06.604468    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"]
Sep 17 19:22:06.607112 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:22:06.607080    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4216b47_a57d_465d_b826_7a485ba64ff2.slice/crio-ffdafe92f2db261003b54b1e399491c37ef684f1c63327a6d1a63b435a5df6c4 WatchSource:0}: Error finding container ffdafe92f2db261003b54b1e399491c37ef684f1c63327a6d1a63b435a5df6c4: Status 404 returned error can't find the container with id ffdafe92f2db261003b54b1e399491c37ef684f1c63327a6d1a63b435a5df6c4
Sep 17 19:22:07.248785 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:07.248745    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:07.299235 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:07.299197    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f" event={"ID":"c4216b47-a57d-465d-b826-7a485ba64ff2","Type":"ContainerStarted","Data":"4fafc9a63fc880d5bdd0b0c183340dc3dbc073b95d31a35604ae119a2d114ffd"}
Sep 17 19:22:07.299422 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:07.299264    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f" event={"ID":"c4216b47-a57d-465d-b826-7a485ba64ff2","Type":"ContainerStarted","Data":"ffdafe92f2db261003b54b1e399491c37ef684f1c63327a6d1a63b435a5df6c4"}
Sep 17 19:22:12.321828 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:12.321799    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f_c4216b47-a57d-465d-b826-7a485ba64ff2/storage-initializer/0.log"
Sep 17 19:22:12.322276 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:12.321836    2567 generic.go:356] "Generic (PLEG): container finished" podID="c4216b47-a57d-465d-b826-7a485ba64ff2" containerID="4fafc9a63fc880d5bdd0b0c183340dc3dbc073b95d31a35604ae119a2d114ffd" exitCode=1
Sep 17 19:22:12.322276 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:12.321908    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f" event={"ID":"c4216b47-a57d-465d-b826-7a485ba64ff2","Type":"ContainerDied","Data":"4fafc9a63fc880d5bdd0b0c183340dc3dbc073b95d31a35604ae119a2d114ffd"}
Sep 17 19:22:13.329666 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:13.329634    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f_c4216b47-a57d-465d-b826-7a485ba64ff2/storage-initializer/0.log"
Sep 17 19:22:13.330105 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:13.329762    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f" event={"ID":"c4216b47-a57d-465d-b826-7a485ba64ff2","Type":"ContainerStarted","Data":"4d9acd4ae0b19d70da1b3f3dcf708fca675f1390da42835897f4a05bdb28a1c4"}
Sep 17 19:22:16.344924 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:16.344845    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f_c4216b47-a57d-465d-b826-7a485ba64ff2/storage-initializer/1.log"
Sep 17 19:22:16.345329 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:16.345295    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f_c4216b47-a57d-465d-b826-7a485ba64ff2/storage-initializer/0.log"
Sep 17 19:22:16.345372 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:16.345332    2567 generic.go:356] "Generic (PLEG): container finished" podID="c4216b47-a57d-465d-b826-7a485ba64ff2" containerID="4d9acd4ae0b19d70da1b3f3dcf708fca675f1390da42835897f4a05bdb28a1c4" exitCode=1
Sep 17 19:22:16.345430 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:16.345407    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f" event={"ID":"c4216b47-a57d-465d-b826-7a485ba64ff2","Type":"ContainerDied","Data":"4d9acd4ae0b19d70da1b3f3dcf708fca675f1390da42835897f4a05bdb28a1c4"}
Sep 17 19:22:16.345492 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:16.345482    2567 scope.go:117] "RemoveContainer" containerID="4fafc9a63fc880d5bdd0b0c183340dc3dbc073b95d31a35604ae119a2d114ffd"
Sep 17 19:22:16.345881 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:16.345863    2567 scope.go:117] "RemoveContainer" containerID="4fafc9a63fc880d5bdd0b0c183340dc3dbc073b95d31a35604ae119a2d114ffd"
Sep 17 19:22:16.356694 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:22:16.356667    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f_kserve-ci-e2e-test(c4216b47-a57d-465d-b826-7a485ba64ff2)\"" pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f" podUID="c4216b47-a57d-465d-b826-7a485ba64ff2"
Sep 17 19:22:16.356904 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:22:16.356874    2567 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f_kserve-ci-e2e-test_c4216b47-a57d-465d-b826-7a485ba64ff2_0 in pod sandbox ffdafe92f2db261003b54b1e399491c37ef684f1c63327a6d1a63b435a5df6c4 from index: no such id: '4fafc9a63fc880d5bdd0b0c183340dc3dbc073b95d31a35604ae119a2d114ffd'" containerID="4fafc9a63fc880d5bdd0b0c183340dc3dbc073b95d31a35604ae119a2d114ffd"
Sep 17 19:22:16.356963 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:16.356919    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fafc9a63fc880d5bdd0b0c183340dc3dbc073b95d31a35604ae119a2d114ffd"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f_kserve-ci-e2e-test_c4216b47-a57d-465d-b826-7a485ba64ff2_0 in pod sandbox ffdafe92f2db261003b54b1e399491c37ef684f1c63327a6d1a63b435a5df6c4 from index: no such id: '4fafc9a63fc880d5bdd0b0c183340dc3dbc073b95d31a35604ae119a2d114ffd'"
Sep 17 19:22:17.351183 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:17.351130    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f_c4216b47-a57d-465d-b826-7a485ba64ff2/storage-initializer/1.log"
Sep 17 19:22:24.043513 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.043473    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"]
Sep 17 19:22:24.129549 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.129483    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:24.179514 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.179488    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f_c4216b47-a57d-465d-b826-7a485ba64ff2/storage-initializer/1.log"
Sep 17 19:22:24.183447 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.183426    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:24.208679 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.208642    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-cabundle-cert\") pod \"c4216b47-a57d-465d-b826-7a485ba64ff2\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") "
Sep 17 19:22:24.208679 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.208687    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4216b47-a57d-465d-b826-7a485ba64ff2-kserve-provision-location\") pod \"c4216b47-a57d-465d-b826-7a485ba64ff2\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") "
Sep 17 19:22:24.208914 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.208731    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4216b47-a57d-465d-b826-7a485ba64ff2-proxy-tls\") pod \"c4216b47-a57d-465d-b826-7a485ba64ff2\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") "
Sep 17 19:22:24.208914 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.208822    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-rbac-proxy-sar-config\") pod \"c4216b47-a57d-465d-b826-7a485ba64ff2\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") "
Sep 17 19:22:24.208914 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.208854    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldqd2\" (UniqueName: \"kubernetes.io/projected/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-api-access-ldqd2\") pod \"c4216b47-a57d-465d-b826-7a485ba64ff2\" (UID: \"c4216b47-a57d-465d-b826-7a485ba64ff2\") "
Sep 17 19:22:24.209072 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.209000    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4216b47-a57d-465d-b826-7a485ba64ff2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4216b47-a57d-465d-b826-7a485ba64ff2" (UID: "c4216b47-a57d-465d-b826-7a485ba64ff2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:24.209122 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.209104    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c4216b47-a57d-465d-b826-7a485ba64ff2" (UID: "c4216b47-a57d-465d-b826-7a485ba64ff2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:24.209223 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.209198    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c4216b47-a57d-465d-b826-7a485ba64ff2" (UID: "c4216b47-a57d-465d-b826-7a485ba64ff2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:24.209289 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.209240    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-cabundle-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:24.209289 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.209263    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4216b47-a57d-465d-b826-7a485ba64ff2-kserve-provision-location\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:24.210157 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.210116    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:24.210948 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.210930    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4216b47-a57d-465d-b826-7a485ba64ff2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4216b47-a57d-465d-b826-7a485ba64ff2" (UID: "c4216b47-a57d-465d-b826-7a485ba64ff2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:22:24.211053 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.211039    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-api-access-ldqd2" (OuterVolumeSpecName: "kube-api-access-ldqd2") pod "c4216b47-a57d-465d-b826-7a485ba64ff2" (UID: "c4216b47-a57d-465d-b826-7a485ba64ff2"). InnerVolumeSpecName "kube-api-access-ldqd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:24.310556 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.310463    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4216b47-a57d-465d-b826-7a485ba64ff2-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:24.310556 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.310497    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:24.310556 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.310509    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldqd2\" (UniqueName: \"kubernetes.io/projected/c4216b47-a57d-465d-b826-7a485ba64ff2-kube-api-access-ldqd2\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:24.381501 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.381466    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f_c4216b47-a57d-465d-b826-7a485ba64ff2/storage-initializer/1.log"
Sep 17 19:22:24.381709 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.381582    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"
Sep 17 19:22:24.381709 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.381594    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f" event={"ID":"c4216b47-a57d-465d-b826-7a485ba64ff2","Type":"ContainerDied","Data":"ffdafe92f2db261003b54b1e399491c37ef684f1c63327a6d1a63b435a5df6c4"}
Sep 17 19:22:24.381709 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.381660    2567 scope.go:117] "RemoveContainer" containerID="4d9acd4ae0b19d70da1b3f3dcf708fca675f1390da42835897f4a05bdb28a1c4"
Sep 17 19:22:24.427095 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.427060    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"]
Sep 17 19:22:24.433828 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:24.433797    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-5ad3e2-predictor-79997bf766-qlh4f"]
Sep 17 19:22:25.083110 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:25.083066    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4216b47-a57d-465d-b826-7a485ba64ff2" path="/var/lib/kubelet/pods/c4216b47-a57d-465d-b826-7a485ba64ff2/volumes"
Sep 17 19:22:26.883981 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:26.883950    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-09f87-predictor-6547b66f58-bhnwg_b0ec727d-e510-410c-bb6a-24a73b423c7c/kserve-container/0.log"
Sep 17 19:22:27.084877 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.084840    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:27.103068 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.103036    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"]
Sep 17 19:22:27.103325 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.103290    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg" podUID="b0ec727d-e510-410c-bb6a-24a73b423c7c" containerName="kserve-container" containerID="cri-o://f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5" gracePeriod=30
Sep 17 19:22:27.103325 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.103315    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg" podUID="b0ec727d-e510-410c-bb6a-24a73b423c7c" containerName="kube-rbac-proxy" containerID="cri-o://dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf" gracePeriod=30
Sep 17 19:22:27.184867 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.184776    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:27.224790 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.224751    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"]
Sep 17 19:22:27.225365 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.225347    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4216b47-a57d-465d-b826-7a485ba64ff2" containerName="storage-initializer"
Sep 17 19:22:27.225412 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.225369    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4216b47-a57d-465d-b826-7a485ba64ff2" containerName="storage-initializer"
Sep 17 19:22:27.225412 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.225380    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4216b47-a57d-465d-b826-7a485ba64ff2" containerName="storage-initializer"
Sep 17 19:22:27.225412 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.225389    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4216b47-a57d-465d-b826-7a485ba64ff2" containerName="storage-initializer"
Sep 17 19:22:27.225501 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.225473    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4216b47-a57d-465d-b826-7a485ba64ff2" containerName="storage-initializer"
Sep 17 19:22:27.225643 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.225632    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4216b47-a57d-465d-b826-7a485ba64ff2" containerName="storage-initializer"
Sep 17 19:22:27.229106 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.229087    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.231736 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.231704    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-bdd76-predictor-serving-cert\""
Sep 17 19:22:27.231871 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.231796    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-bdd76-kube-rbac-proxy-sar-config\""
Sep 17 19:22:27.232400 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.232377    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:22:27.232636 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.232619    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:22:27.241391 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.241361    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"]
Sep 17 19:22:27.261486 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.261453    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:27.334778 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.334741    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61be52ca-515b-4302-baac-f9350a9340cc-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.334951 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.334833    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-cabundle-cert\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.334951 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.334904    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61be52ca-515b-4302-baac-f9350a9340cc-proxy-tls\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.334951 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.334941    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.335067 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.334970    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl72m\" (UniqueName: \"kubernetes.io/projected/61be52ca-515b-4302-baac-f9350a9340cc-kube-api-access-gl72m\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.354534 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.354509    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:22:27.396239 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.396201    2567 generic.go:356] "Generic (PLEG): container finished" podID="b0ec727d-e510-410c-bb6a-24a73b423c7c" containerID="dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf" exitCode=2
Sep 17 19:22:27.396239 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.396236    2567 generic.go:356] "Generic (PLEG): container finished" podID="b0ec727d-e510-410c-bb6a-24a73b423c7c" containerID="f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5" exitCode=2
Sep 17 19:22:27.396462 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.396259    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"
Sep 17 19:22:27.396462 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.396275    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg" event={"ID":"b0ec727d-e510-410c-bb6a-24a73b423c7c","Type":"ContainerDied","Data":"dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf"}
Sep 17 19:22:27.396462 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.396338    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg" event={"ID":"b0ec727d-e510-410c-bb6a-24a73b423c7c","Type":"ContainerDied","Data":"f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5"}
Sep 17 19:22:27.396462 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.396369    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg" event={"ID":"b0ec727d-e510-410c-bb6a-24a73b423c7c","Type":"ContainerDied","Data":"f85e4d38a4692cb8c6458c6f4d778a4b34e326ec7a0bfa28f4343f9d0cbe8ee1"}
Sep 17 19:22:27.396462 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.396406    2567 scope.go:117] "RemoveContainer" containerID="dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf"
Sep 17 19:22:27.404625 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.404607    2567 scope.go:117] "RemoveContainer" containerID="f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5"
Sep 17 19:22:27.412751 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.412729    2567 scope.go:117] "RemoveContainer" containerID="dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf"
Sep 17 19:22:27.413082 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:22:27.413061    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf\": container with ID starting with dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf not found: ID does not exist" containerID="dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf"
Sep 17 19:22:27.413159 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.413093    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf"} err="failed to get container status \"dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf\": rpc error: code = NotFound desc = could not find container \"dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf\": container with ID starting with dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf not found: ID does not exist"
Sep 17 19:22:27.413159 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.413121    2567 scope.go:117] "RemoveContainer" containerID="f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5"
Sep 17 19:22:27.413412 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:22:27.413386    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5\": container with ID starting with f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5 not found: ID does not exist" containerID="f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5"
Sep 17 19:22:27.413459 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.413413    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5"} err="failed to get container status \"f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5\": rpc error: code = NotFound desc = could not find container \"f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5\": container with ID starting with f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5 not found: ID does not exist"
Sep 17 19:22:27.413459 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.413435    2567 scope.go:117] "RemoveContainer" containerID="dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf"
Sep 17 19:22:27.413656 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.413638    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf"} err="failed to get container status \"dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf\": rpc error: code = NotFound desc = could not find container \"dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf\": container with ID starting with dc407c507bdc078adeae7d207dba6609be29c2271aa83b433521e26b208d4ccf not found: ID does not exist"
Sep 17 19:22:27.413694 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.413657    2567 scope.go:117] "RemoveContainer" containerID="f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5"
Sep 17 19:22:27.413919 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.413895    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5"} err="failed to get container status \"f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5\": rpc error: code = NotFound desc = could not find container \"f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5\": container with ID starting with f02c000722ba3693f009b25efb6260a6b42edf9d82508773e487cc4d5c7379a5 not found: ID does not exist"
Sep 17 19:22:27.435685 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.435583    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0ec727d-e510-410c-bb6a-24a73b423c7c-proxy-tls\") pod \"b0ec727d-e510-410c-bb6a-24a73b423c7c\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") "
Sep 17 19:22:27.435685 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.435631    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-rbac-proxy-sar-config\") pod \"b0ec727d-e510-410c-bb6a-24a73b423c7c\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") "
Sep 17 19:22:27.435685 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.435653    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwk2b\" (UniqueName: \"kubernetes.io/projected/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-api-access-gwk2b\") pod \"b0ec727d-e510-410c-bb6a-24a73b423c7c\" (UID: \"b0ec727d-e510-410c-bb6a-24a73b423c7c\") "
Sep 17 19:22:27.435965 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.435755    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-cabundle-cert\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.435965 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.435796    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61be52ca-515b-4302-baac-f9350a9340cc-proxy-tls\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.435965 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.435817    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.435965 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.435845    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl72m\" (UniqueName: \"kubernetes.io/projected/61be52ca-515b-4302-baac-f9350a9340cc-kube-api-access-gl72m\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.435965 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.435896    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61be52ca-515b-4302-baac-f9350a9340cc-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.436546 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.436513    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0ec727d-e510-410c-bb6a-24a73b423c7c" (UID: "b0ec727d-e510-410c-bb6a-24a73b423c7c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:27.436642 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.436565    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-cabundle-cert\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.436642 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.436565    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.436811 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.436788    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61be52ca-515b-4302-baac-f9350a9340cc-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.437994 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.437966    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-api-access-gwk2b" (OuterVolumeSpecName: "kube-api-access-gwk2b") pod "b0ec727d-e510-410c-bb6a-24a73b423c7c" (UID: "b0ec727d-e510-410c-bb6a-24a73b423c7c"). InnerVolumeSpecName "kube-api-access-gwk2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:27.438084 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.438036    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ec727d-e510-410c-bb6a-24a73b423c7c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0ec727d-e510-410c-bb6a-24a73b423c7c" (UID: "b0ec727d-e510-410c-bb6a-24a73b423c7c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:22:27.438404 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.438388    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61be52ca-515b-4302-baac-f9350a9340cc-proxy-tls\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.447023 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.447000    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl72m\" (UniqueName: \"kubernetes.io/projected/61be52ca-515b-4302-baac-f9350a9340cc-kube-api-access-gl72m\") pod \"isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.480555 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.480513    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:27.537298 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.537247    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0ec727d-e510-410c-bb6a-24a73b423c7c-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:27.537298 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.537291    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:27.537298 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.537301    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwk2b\" (UniqueName: \"kubernetes.io/projected/b0ec727d-e510-410c-bb6a-24a73b423c7c-kube-api-access-gwk2b\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:27.540097 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.540072    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:27.718754 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.718717    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"]
Sep 17 19:22:27.723400 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.723374    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-09f87-predictor-6547b66f58-bhnwg"]
Sep 17 19:22:27.816070 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:27.815933    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"]
Sep 17 19:22:27.818348 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:22:27.818320    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61be52ca_515b_4302_baac_f9350a9340cc.slice/crio-bc9cdc63eb762aae4d3d278b0876b075c3b57856ea0c8dda65870e9ca235bae4 WatchSource:0}: Error finding container bc9cdc63eb762aae4d3d278b0876b075c3b57856ea0c8dda65870e9ca235bae4: Status 404 returned error can't find the container with id bc9cdc63eb762aae4d3d278b0876b075c3b57856ea0c8dda65870e9ca235bae4
Sep 17 19:22:28.403679 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:28.403644    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" event={"ID":"61be52ca-515b-4302-baac-f9350a9340cc","Type":"ContainerStarted","Data":"74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c"}
Sep 17 19:22:28.404206 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:28.403702    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" event={"ID":"61be52ca-515b-4302-baac-f9350a9340cc","Type":"ContainerStarted","Data":"bc9cdc63eb762aae4d3d278b0876b075c3b57856ea0c8dda65870e9ca235bae4"}
Sep 17 19:22:29.088674 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.088637    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ec727d-e510-410c-bb6a-24a73b423c7c" path="/var/lib/kubelet/pods/b0ec727d-e510-410c-bb6a-24a73b423c7c/volumes"
Sep 17 19:22:29.226228 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.226179    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:29.255500 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.255459    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"]
Sep 17 19:22:29.255899 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.255879    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ec727d-e510-410c-bb6a-24a73b423c7c" containerName="kserve-container"
Sep 17 19:22:29.256014 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.255901    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ec727d-e510-410c-bb6a-24a73b423c7c" containerName="kserve-container"
Sep 17 19:22:29.256014 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.255942    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ec727d-e510-410c-bb6a-24a73b423c7c" containerName="kube-rbac-proxy"
Sep 17 19:22:29.256014 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.255953    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ec727d-e510-410c-bb6a-24a73b423c7c" containerName="kube-rbac-proxy"
Sep 17 19:22:29.256213 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.256039    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ec727d-e510-410c-bb6a-24a73b423c7c" containerName="kube-rbac-proxy"
Sep 17 19:22:29.256213 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.256058    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ec727d-e510-410c-bb6a-24a73b423c7c" containerName="kserve-container"
Sep 17 19:22:29.259589 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.259566    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.262194 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.262167    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-44ebcf\""
Sep 17 19:22:29.262194 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.262187    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-44ebcf-dockercfg-59xtg\""
Sep 17 19:22:29.262354 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.262170    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-44ebcf-kube-rbac-proxy-sar-config\""
Sep 17 19:22:29.262354 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.262248    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-44ebcf-predictor-serving-cert\""
Sep 17 19:22:29.269094 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.269068    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"]
Sep 17 19:22:29.352072 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.351981    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-proxy-tls\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.352072 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.352017    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.352072 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.352065    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kserve-provision-location\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.352360 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.352080    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-cabundle-cert\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.352360 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.352186    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52bdm\" (UniqueName: \"kubernetes.io/projected/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-api-access-52bdm\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.453493 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.453448    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-proxy-tls\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.453493 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.453489    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.454022 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.453568    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kserve-provision-location\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.454022 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.453594    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-cabundle-cert\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.454022 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:22:29.453619    2567 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-serving-cert: secret "isvc-init-fail-44ebcf-predictor-serving-cert" not found
Sep 17 19:22:29.454022 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.453651    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52bdm\" (UniqueName: \"kubernetes.io/projected/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-api-access-52bdm\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.454022 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:22:29.453692    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-proxy-tls podName:ed8ce029-5d24-4980-bee7-d3bf3cfb001e nodeName:}" failed. No retries permitted until 2026-09-17 19:22:29.953667949 +0000 UTC m=+972.879886362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-proxy-tls") pod "isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh" (UID: "ed8ce029-5d24-4980-bee7-d3bf3cfb001e") : secret "isvc-init-fail-44ebcf-predictor-serving-cert" not found
Sep 17 19:22:29.454229 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.454094    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kserve-provision-location\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.454273 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.454250    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.454480 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.454354    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-cabundle-cert\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.462868 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.462842    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52bdm\" (UniqueName: \"kubernetes.io/projected/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-api-access-52bdm\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.745689 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.745649    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:29.957778 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.957742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-proxy-tls\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:29.960243 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:29.960212    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-proxy-tls\") pod \"isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") " pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:30.172023 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:30.171984    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:30.448162 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:30.448121    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"]
Sep 17 19:22:30.450540 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:22:30.450496    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8ce029_5d24_4980_bee7_d3bf3cfb001e.slice/crio-9b2af671ff52641181846c0a9e1e0e6e63eb37263a700ef1fa6e877b9bd53341 WatchSource:0}: Error finding container 9b2af671ff52641181846c0a9e1e0e6e63eb37263a700ef1fa6e877b9bd53341: Status 404 returned error can't find the container with id 9b2af671ff52641181846c0a9e1e0e6e63eb37263a700ef1fa6e877b9bd53341
Sep 17 19:22:31.420614 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:31.420532    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh" event={"ID":"ed8ce029-5d24-4980-bee7-d3bf3cfb001e","Type":"ContainerStarted","Data":"a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a"}
Sep 17 19:22:31.420614 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:31.420591    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh" event={"ID":"ed8ce029-5d24-4980-bee7-d3bf3cfb001e","Type":"ContainerStarted","Data":"9b2af671ff52641181846c0a9e1e0e6e63eb37263a700ef1fa6e877b9bd53341"}
Sep 17 19:22:32.426235 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:32.426188    2567 generic.go:356] "Generic (PLEG): container finished" podID="61be52ca-515b-4302-baac-f9350a9340cc" containerID="74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c" exitCode=0
Sep 17 19:22:32.426634 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:32.426270    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" event={"ID":"61be52ca-515b-4302-baac-f9350a9340cc","Type":"ContainerDied","Data":"74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c"}
Sep 17 19:22:33.433438 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:33.433399    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" event={"ID":"61be52ca-515b-4302-baac-f9350a9340cc","Type":"ContainerStarted","Data":"60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5"}
Sep 17 19:22:33.433831 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:33.433497    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" event={"ID":"61be52ca-515b-4302-baac-f9350a9340cc","Type":"ContainerStarted","Data":"3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329"}
Sep 17 19:22:33.433831 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:33.433701    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:33.458100 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:33.458044    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podStartSLOduration=2.486088512 podStartE2EDuration="6.458026915s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.971938403s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:22:33.455468671 +0000 UTC m=+976.381687111" watchObservedRunningTime="2026-09-17 19:22:33.458026915 +0000 UTC m=+976.384245337"
Sep 17 19:22:34.438224 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:34.438175    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:34.439704 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:34.439676    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:22:35.442514 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:35.442469    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:22:36.448299 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:36.448272    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh_ed8ce029-5d24-4980-bee7-d3bf3cfb001e/storage-initializer/0.log"
Sep 17 19:22:36.448677 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:36.448309    2567 generic.go:356] "Generic (PLEG): container finished" podID="ed8ce029-5d24-4980-bee7-d3bf3cfb001e" containerID="a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a" exitCode=1
Sep 17 19:22:36.448677 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:36.448351    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh" event={"ID":"ed8ce029-5d24-4980-bee7-d3bf3cfb001e","Type":"ContainerDied","Data":"a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a"}
Sep 17 19:22:37.455283 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:37.455252    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh_ed8ce029-5d24-4980-bee7-d3bf3cfb001e/storage-initializer/0.log"
Sep 17 19:22:37.455678 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:37.455320    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh" event={"ID":"ed8ce029-5d24-4980-bee7-d3bf3cfb001e","Type":"ContainerStarted","Data":"c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0"}
Sep 17 19:22:39.242969 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:39.242927    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"]
Sep 17 19:22:39.243495 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:39.243459    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh" podUID="ed8ce029-5d24-4980-bee7-d3bf3cfb001e" containerName="storage-initializer" containerID="cri-o://c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0" gracePeriod=30
Sep 17 19:22:39.254583 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:39.254555    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:40.447472 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:40.447441    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:22:40.448217 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:40.448184    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:22:40.998632 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:40.998607    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh_ed8ce029-5d24-4980-bee7-d3bf3cfb001e/storage-initializer/1.log"
Sep 17 19:22:40.999003 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:40.998989    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh_ed8ce029-5d24-4980-bee7-d3bf3cfb001e/storage-initializer/0.log"
Sep 17 19:22:41.001376 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.001360    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:41.062310 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.062273    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-rbac-proxy-sar-config\") pod \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") "
Sep 17 19:22:41.062310 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.062326    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-proxy-tls\") pod \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") "
Sep 17 19:22:41.062586 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.062351    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52bdm\" (UniqueName: \"kubernetes.io/projected/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-api-access-52bdm\") pod \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") "
Sep 17 19:22:41.062586 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.062426    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-cabundle-cert\") pod \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") "
Sep 17 19:22:41.062586 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.062453    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kserve-provision-location\") pod \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\" (UID: \"ed8ce029-5d24-4980-bee7-d3bf3cfb001e\") "
Sep 17 19:22:41.062787 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.062717    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed8ce029-5d24-4980-bee7-d3bf3cfb001e" (UID: "ed8ce029-5d24-4980-bee7-d3bf3cfb001e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:41.062828 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.062800    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed8ce029-5d24-4980-bee7-d3bf3cfb001e" (UID: "ed8ce029-5d24-4980-bee7-d3bf3cfb001e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:22:41.062860 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.062831    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed8ce029-5d24-4980-bee7-d3bf3cfb001e" (UID: "ed8ce029-5d24-4980-bee7-d3bf3cfb001e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:22:41.064583 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.064563    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed8ce029-5d24-4980-bee7-d3bf3cfb001e" (UID: "ed8ce029-5d24-4980-bee7-d3bf3cfb001e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:22:41.064695 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.064676    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-api-access-52bdm" (OuterVolumeSpecName: "kube-api-access-52bdm") pod "ed8ce029-5d24-4980-bee7-d3bf3cfb001e" (UID: "ed8ce029-5d24-4980-bee7-d3bf3cfb001e"). InnerVolumeSpecName "kube-api-access-52bdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:22:41.163322 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.163278    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-cabundle-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:41.163322 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.163315    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kserve-provision-location\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:41.163322 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.163326    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:41.163322 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.163335    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:41.163604 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.163346    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52bdm\" (UniqueName: \"kubernetes.io/projected/ed8ce029-5d24-4980-bee7-d3bf3cfb001e-kube-api-access-52bdm\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:22:41.473274 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.473196    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh_ed8ce029-5d24-4980-bee7-d3bf3cfb001e/storage-initializer/1.log"
Sep 17 19:22:41.473655 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.473606    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh_ed8ce029-5d24-4980-bee7-d3bf3cfb001e/storage-initializer/0.log"
Sep 17 19:22:41.473655 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.473640    2567 generic.go:356] "Generic (PLEG): container finished" podID="ed8ce029-5d24-4980-bee7-d3bf3cfb001e" containerID="c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0" exitCode=1
Sep 17 19:22:41.473733 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.473694    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"
Sep 17 19:22:41.473733 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.473708    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh" event={"ID":"ed8ce029-5d24-4980-bee7-d3bf3cfb001e","Type":"ContainerDied","Data":"c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0"}
Sep 17 19:22:41.473802 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.473764    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh" event={"ID":"ed8ce029-5d24-4980-bee7-d3bf3cfb001e","Type":"ContainerDied","Data":"9b2af671ff52641181846c0a9e1e0e6e63eb37263a700ef1fa6e877b9bd53341"}
Sep 17 19:22:41.473802 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.473791    2567 scope.go:117] "RemoveContainer" containerID="c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0"
Sep 17 19:22:41.482337 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.482321    2567 scope.go:117] "RemoveContainer" containerID="a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a"
Sep 17 19:22:41.490228 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.490198    2567 scope.go:117] "RemoveContainer" containerID="c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0"
Sep 17 19:22:41.490531 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:22:41.490511    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0\": container with ID starting with c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0 not found: ID does not exist" containerID="c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0"
Sep 17 19:22:41.490604 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.490540    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0"} err="failed to get container status \"c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0\": rpc error: code = NotFound desc = could not find container \"c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0\": container with ID starting with c98cee61000ec28935ab861014884f7f936763a380876f0755b6e21d21315cf0 not found: ID does not exist"
Sep 17 19:22:41.490604 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.490560    2567 scope.go:117] "RemoveContainer" containerID="a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a"
Sep 17 19:22:41.490845 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:22:41.490820    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a\": container with ID starting with a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a not found: ID does not exist" containerID="a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a"
Sep 17 19:22:41.490886 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.490853    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a"} err="failed to get container status \"a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a\": rpc error: code = NotFound desc = could not find container \"a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a\": container with ID starting with a8272884899b08c889c82f8cf2fc2f1edec5e2f1cae6b0bd5640339c1b488b4a not found: ID does not exist"
Sep 17 19:22:41.507688 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.507661    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"]
Sep 17 19:22:41.511316 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:41.511291    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-44ebcf-predictor-567c7f4f5b-lwmfh"]
Sep 17 19:22:43.082877 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:43.082843    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed8ce029-5d24-4980-bee7-d3bf3cfb001e" path="/var/lib/kubelet/pods/ed8ce029-5d24-4980-bee7-d3bf3cfb001e/volumes"
Sep 17 19:22:44.473153 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.473095    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:44.499462 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.499410    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"]
Sep 17 19:22:44.500157 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.500111    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed8ce029-5d24-4980-bee7-d3bf3cfb001e" containerName="storage-initializer"
Sep 17 19:22:44.500157 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.500152    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed8ce029-5d24-4980-bee7-d3bf3cfb001e" containerName="storage-initializer"
Sep 17 19:22:44.500328 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.500195    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed8ce029-5d24-4980-bee7-d3bf3cfb001e" containerName="storage-initializer"
Sep 17 19:22:44.500328 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.500204    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed8ce029-5d24-4980-bee7-d3bf3cfb001e" containerName="storage-initializer"
Sep 17 19:22:44.500429 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.500356    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed8ce029-5d24-4980-bee7-d3bf3cfb001e" containerName="storage-initializer"
Sep 17 19:22:44.500429 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.500375    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed8ce029-5d24-4980-bee7-d3bf3cfb001e" containerName="storage-initializer"
Sep 17 19:22:44.506012 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.505983    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.508920 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.508896    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-b0e4a-kube-rbac-proxy-sar-config\""
Sep 17 19:22:44.508920 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.508894    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-b0e4a-predictor-serving-cert\""
Sep 17 19:22:44.514020 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.513996    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"]
Sep 17 19:22:44.593922 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.593889    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b565da1-cef8-43a7-8496-c7362453b442-proxy-tls\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.594122 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.593927    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-cabundle-cert\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.594122 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.593958    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnwt\" (UniqueName: \"kubernetes.io/projected/8b565da1-cef8-43a7-8496-c7362453b442-kube-api-access-wfnwt\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.594122 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.594070    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.594122 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.594118    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b565da1-cef8-43a7-8496-c7362453b442-kserve-provision-location\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.657804 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.657769    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:44.694845 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.694805    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnwt\" (UniqueName: \"kubernetes.io/projected/8b565da1-cef8-43a7-8496-c7362453b442-kube-api-access-wfnwt\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.695023 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.694897    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.695023 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.694944    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b565da1-cef8-43a7-8496-c7362453b442-kserve-provision-location\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.695023 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.694983    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b565da1-cef8-43a7-8496-c7362453b442-proxy-tls\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.695023 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.695007    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-cabundle-cert\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.695484 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.695462    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b565da1-cef8-43a7-8496-c7362453b442-kserve-provision-location\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.695653 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.695634    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-cabundle-cert\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.695728 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.695708    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.697551 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.697526    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b565da1-cef8-43a7-8496-c7362453b442-proxy-tls\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.704955 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.704925    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnwt\" (UniqueName: \"kubernetes.io/projected/8b565da1-cef8-43a7-8496-c7362453b442-kube-api-access-wfnwt\") pod \"raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") " pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:44.817199 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:44.817087    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:45.092368 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:45.092340    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"]
Sep 17 19:22:45.094792 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:22:45.094759    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b565da1_cef8_43a7_8496_c7362453b442.slice/crio-5716ad7ce32b7e2af95b50719bcbff1a4c195dbc2530cddc7e150711226f43f5 WatchSource:0}: Error finding container 5716ad7ce32b7e2af95b50719bcbff1a4c195dbc2530cddc7e150711226f43f5: Status 404 returned error can't find the container with id 5716ad7ce32b7e2af95b50719bcbff1a4c195dbc2530cddc7e150711226f43f5
Sep 17 19:22:45.493484 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:45.493442    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" event={"ID":"8b565da1-cef8-43a7-8496-c7362453b442","Type":"ContainerStarted","Data":"059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1"}
Sep 17 19:22:45.493879 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:45.493519    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" event={"ID":"8b565da1-cef8-43a7-8496-c7362453b442","Type":"ContainerStarted","Data":"5716ad7ce32b7e2af95b50719bcbff1a4c195dbc2530cddc7e150711226f43f5"}
Sep 17 19:22:49.512860 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:49.512826    2567 generic.go:356] "Generic (PLEG): container finished" podID="8b565da1-cef8-43a7-8496-c7362453b442" containerID="059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1" exitCode=0
Sep 17 19:22:49.513276 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:49.512880    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" event={"ID":"8b565da1-cef8-43a7-8496-c7362453b442","Type":"ContainerDied","Data":"059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1"}
Sep 17 19:22:50.448525 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:50.448486    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:22:50.520017 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:50.519983    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" event={"ID":"8b565da1-cef8-43a7-8496-c7362453b442","Type":"ContainerStarted","Data":"43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c"}
Sep 17 19:22:50.520469 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:50.520044    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" event={"ID":"8b565da1-cef8-43a7-8496-c7362453b442","Type":"ContainerStarted","Data":"b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668"}
Sep 17 19:22:50.520469 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:50.520355    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:50.520581 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:50.520494    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:50.521945 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:50.521915    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:22:50.547928 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:50.547878    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podStartSLOduration=2.671686739 podStartE2EDuration="6.547859594s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.876172855s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:22:50.545582005 +0000 UTC m=+993.471800428" watchObservedRunningTime="2026-09-17 19:22:50.547859594 +0000 UTC m=+993.474078017"
Sep 17 19:22:51.525005 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:51.524968    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:22:56.530448 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:56.530416    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:22:56.530968 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:22:56.530937    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:23:00.448369 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:00.448323    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:23:02.275973 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:02.275939    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:06.530953 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:06.530906    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:23:10.448564 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:10.448523    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:23:16.531445 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:16.531397    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:23:20.448911 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:20.448866    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:23:26.531109 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:26.531059    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:23:30.448727 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:30.448688    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:23:36.531562 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:36.531513    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:23:40.448114 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:40.448072    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:23:42.078403 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:42.078360    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:23:46.531113 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:46.531072    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:23:52.079335 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:52.079276    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:23:56.336805 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:56.336767    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:56.531116 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:23:56.531068    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:24:02.078768 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:02.078676    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:24:06.531904 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:06.531854    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:24:09.080091 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:09.080052    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:24:12.078908 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:12.078860    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:24:19.083489 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:19.083460    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:24:22.078961 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:22.078915    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:24:25.109122 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:25.109076    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"]
Sep 17 19:24:25.109563 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:25.109433    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" containerID="cri-o://b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668" gracePeriod=30
Sep 17 19:24:25.109563 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:25.109494    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kube-rbac-proxy" containerID="cri-o://43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c" gracePeriod=30
Sep 17 19:24:25.131553 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:25.131521    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:25.737882 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:25.737842    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:25.934340 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:25.934302    2567 generic.go:356] "Generic (PLEG): container finished" podID="8b565da1-cef8-43a7-8496-c7362453b442" containerID="43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c" exitCode=2
Sep 17 19:24:25.934526 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:25.934377    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" event={"ID":"8b565da1-cef8-43a7-8496-c7362453b442","Type":"ContainerDied","Data":"43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c"}
Sep 17 19:24:26.525492 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:26.525447    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 19:24:29.080278 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:29.080238    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:24:30.218315 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.218277    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:30.301294 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.301257    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:30.763043 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.763013    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:24:30.818439 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.818346    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b565da1-cef8-43a7-8496-c7362453b442-proxy-tls\") pod \"8b565da1-cef8-43a7-8496-c7362453b442\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") "
Sep 17 19:24:30.818439 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.818391    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-cabundle-cert\") pod \"8b565da1-cef8-43a7-8496-c7362453b442\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") "
Sep 17 19:24:30.818439 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.818411    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b565da1-cef8-43a7-8496-c7362453b442-kserve-provision-location\") pod \"8b565da1-cef8-43a7-8496-c7362453b442\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") "
Sep 17 19:24:30.818740 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.818544    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfnwt\" (UniqueName: \"kubernetes.io/projected/8b565da1-cef8-43a7-8496-c7362453b442-kube-api-access-wfnwt\") pod \"8b565da1-cef8-43a7-8496-c7362453b442\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") "
Sep 17 19:24:30.818740 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.818720    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-kube-rbac-proxy-sar-config\") pod \"8b565da1-cef8-43a7-8496-c7362453b442\" (UID: \"8b565da1-cef8-43a7-8496-c7362453b442\") "
Sep 17 19:24:30.818870 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.818762    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b565da1-cef8-43a7-8496-c7362453b442-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b565da1-cef8-43a7-8496-c7362453b442" (UID: "8b565da1-cef8-43a7-8496-c7362453b442"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:24:30.818979 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.818952    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b565da1-cef8-43a7-8496-c7362453b442" (UID: "8b565da1-cef8-43a7-8496-c7362453b442"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:24:30.819023 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.818989    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b565da1-cef8-43a7-8496-c7362453b442-kserve-provision-location\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:24:30.819065 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.819033    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b565da1-cef8-43a7-8496-c7362453b442" (UID: "8b565da1-cef8-43a7-8496-c7362453b442"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:24:30.820633 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.820610    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b565da1-cef8-43a7-8496-c7362453b442-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b565da1-cef8-43a7-8496-c7362453b442" (UID: "8b565da1-cef8-43a7-8496-c7362453b442"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:24:30.820747 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.820647    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b565da1-cef8-43a7-8496-c7362453b442-kube-api-access-wfnwt" (OuterVolumeSpecName: "kube-api-access-wfnwt") pod "8b565da1-cef8-43a7-8496-c7362453b442" (UID: "8b565da1-cef8-43a7-8496-c7362453b442"). InnerVolumeSpecName "kube-api-access-wfnwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:24:30.920294 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.920251    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b565da1-cef8-43a7-8496-c7362453b442-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:24:30.920294 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.920285    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-cabundle-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:24:30.920294 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.920295    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfnwt\" (UniqueName: \"kubernetes.io/projected/8b565da1-cef8-43a7-8496-c7362453b442-kube-api-access-wfnwt\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:24:30.920294 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.920305    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b565da1-cef8-43a7-8496-c7362453b442-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:24:30.956275 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.956240    2567 generic.go:356] "Generic (PLEG): container finished" podID="8b565da1-cef8-43a7-8496-c7362453b442" containerID="b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668" exitCode=0
Sep 17 19:24:30.956428 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.956315    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"
Sep 17 19:24:30.956428 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.956338    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" event={"ID":"8b565da1-cef8-43a7-8496-c7362453b442","Type":"ContainerDied","Data":"b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668"}
Sep 17 19:24:30.956502 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.956433    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r" event={"ID":"8b565da1-cef8-43a7-8496-c7362453b442","Type":"ContainerDied","Data":"5716ad7ce32b7e2af95b50719bcbff1a4c195dbc2530cddc7e150711226f43f5"}
Sep 17 19:24:30.956502 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.956461    2567 scope.go:117] "RemoveContainer" containerID="43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c"
Sep 17 19:24:30.965547 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.965526    2567 scope.go:117] "RemoveContainer" containerID="b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668"
Sep 17 19:24:30.973669 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.973644    2567 scope.go:117] "RemoveContainer" containerID="059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1"
Sep 17 19:24:30.983760 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.983738    2567 scope.go:117] "RemoveContainer" containerID="43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c"
Sep 17 19:24:30.984101 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:24:30.984078    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c\": container with ID starting with 43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c not found: ID does not exist" containerID="43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c"
Sep 17 19:24:30.984177 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.984111    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c"} err="failed to get container status \"43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c\": rpc error: code = NotFound desc = could not find container \"43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c\": container with ID starting with 43e90ec64b538c7ce04235391b421bc807f12afdc9040082258de0bec2499a3c not found: ID does not exist"
Sep 17 19:24:30.984177 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.984130    2567 scope.go:117] "RemoveContainer" containerID="b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668"
Sep 17 19:24:30.984466 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:24:30.984448    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668\": container with ID starting with b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668 not found: ID does not exist" containerID="b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668"
Sep 17 19:24:30.984555 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.984473    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668"} err="failed to get container status \"b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668\": rpc error: code = NotFound desc = could not find container \"b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668\": container with ID starting with b380ee63ed2162533e2e41306119d2aa2f2f5c9e042f246708c175c656c83668 not found: ID does not exist"
Sep 17 19:24:30.984555 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.984494    2567 scope.go:117] "RemoveContainer" containerID="059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1"
Sep 17 19:24:30.984743 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:24:30.984725    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1\": container with ID starting with 059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1 not found: ID does not exist" containerID="059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1"
Sep 17 19:24:30.984787 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.984746    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1"} err="failed to get container status \"059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1\": rpc error: code = NotFound desc = could not find container \"059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1\": container with ID starting with 059bef3e2a0c887d6ab9150bca18c828e38c2bd0c863324a818b3ce912d403f1 not found: ID does not exist"
Sep 17 19:24:30.995363 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.995333    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"]
Sep 17 19:24:30.999099 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:30.999078    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-b0e4a-predictor-d99b7bc64-jft2r"]
Sep 17 19:24:31.082632 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:31.082556    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b565da1-cef8-43a7-8496-c7362453b442" path="/var/lib/kubelet/pods/8b565da1-cef8-43a7-8496-c7362453b442/volumes"
Sep 17 19:24:32.079061 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:32.079024    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:24:42.078945 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:42.078900    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:24:52.079011 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:52.078957    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:24:57.077721 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:24:57.077678    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:25:07.078121 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:07.078083    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:25:17.078052 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:17.078001    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:25:27.082785 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:27.082753    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:25:38.514916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:38.514833    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"]
Sep 17 19:25:38.515380 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:38.515122    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" containerID="cri-o://3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329" gracePeriod=30
Sep 17 19:25:38.515380 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:38.515130    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kube-rbac-proxy" containerID="cri-o://60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5" gracePeriod=30
Sep 17 19:25:38.541487 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:38.541453    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:39.235151 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:39.235115    2567 generic.go:356] "Generic (PLEG): container finished" podID="61be52ca-515b-4302-baac-f9350a9340cc" containerID="60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5" exitCode=2
Sep 17 19:25:39.235334 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:39.235189    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" event={"ID":"61be52ca-515b-4302-baac-f9350a9340cc","Type":"ContainerDied","Data":"60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5"}
Sep 17 19:25:40.443215 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:40.443171    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 17 19:25:43.611505 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.611470    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:43.687464 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.687427    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"]
Sep 17 19:25:43.687762 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.687750    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kube-rbac-proxy"
Sep 17 19:25:43.687817 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.687764    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kube-rbac-proxy"
Sep 17 19:25:43.687817 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.687785    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="storage-initializer"
Sep 17 19:25:43.687817 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.687792    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="storage-initializer"
Sep 17 19:25:43.687817 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.687799    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container"
Sep 17 19:25:43.687817 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.687805    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container"
Sep 17 19:25:43.687989 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.687864    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kserve-container"
Sep 17 19:25:43.687989 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.687876    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b565da1-cef8-43a7-8496-c7362453b442" containerName="kube-rbac-proxy"
Sep 17 19:25:43.691102 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.691080    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.693833 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.693806    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 19:25:43.693976 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.693864    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 19:25:43.695461 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.695429    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:43.700746 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.700716    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"]
Sep 17 19:25:43.823174 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.823127    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:43.857952 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.857913    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/306f8a7c-69a7-483e-91ad-3d5295d5b514-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.857952 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.857950    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwxt\" (UniqueName: \"kubernetes.io/projected/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-api-access-rfwxt\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.858223 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.858031    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.858223 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.858076    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.858223 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.858104    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/306f8a7c-69a7-483e-91ad-3d5295d5b514-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.959544 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.959448    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/306f8a7c-69a7-483e-91ad-3d5295d5b514-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.959544 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.959502    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/306f8a7c-69a7-483e-91ad-3d5295d5b514-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.959544 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.959524    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwxt\" (UniqueName: \"kubernetes.io/projected/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-api-access-rfwxt\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.959850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.959562    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.959850 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.959604    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.959964 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.959886    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/306f8a7c-69a7-483e-91ad-3d5295d5b514-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.960327 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.960303    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.960414 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.960309    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.961971 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.961946    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/306f8a7c-69a7-483e-91ad-3d5295d5b514-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:43.968017 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:43.967989    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwxt\" (UniqueName: \"kubernetes.io/projected/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-api-access-rfwxt\") pod \"isvc-canary-create-predictor-7c7f45c769-wfcdh\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:44.003049 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:44.003003    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:44.024400 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:44.024363    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:44.280322 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:44.280294    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"]
Sep 17 19:25:44.282983 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:25:44.282952    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306f8a7c_69a7_483e_91ad_3d5295d5b514.slice/crio-4dcf760462486558acbd66d8cc4bb8ea52e9f6603ff2abea06ba75cf12ad4d30 WatchSource:0}: Error finding container 4dcf760462486558acbd66d8cc4bb8ea52e9f6603ff2abea06ba75cf12ad4d30: Status 404 returned error can't find the container with id 4dcf760462486558acbd66d8cc4bb8ea52e9f6603ff2abea06ba75cf12ad4d30
Sep 17 19:25:45.260903 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:45.260862    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" event={"ID":"306f8a7c-69a7-483e-91ad-3d5295d5b514","Type":"ContainerStarted","Data":"6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35"}
Sep 17 19:25:45.261337 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:45.260922    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" event={"ID":"306f8a7c-69a7-483e-91ad-3d5295d5b514","Type":"ContainerStarted","Data":"4dcf760462486558acbd66d8cc4bb8ea52e9f6603ff2abea06ba75cf12ad4d30"}
Sep 17 19:25:45.443314 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:45.443273    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 17 19:25:47.078115 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:47.078067    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 19:25:49.279284 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:49.279249    2567 generic.go:356] "Generic (PLEG): container finished" podID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerID="6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35" exitCode=0
Sep 17 19:25:49.279680 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:49.279295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" event={"ID":"306f8a7c-69a7-483e-91ad-3d5295d5b514","Type":"ContainerDied","Data":"6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35"}
Sep 17 19:25:49.858568 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:49.858542    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:25:50.015457 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.015363    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-cabundle-cert\") pod \"61be52ca-515b-4302-baac-f9350a9340cc\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") "
Sep 17 19:25:50.015457 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.015410    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl72m\" (UniqueName: \"kubernetes.io/projected/61be52ca-515b-4302-baac-f9350a9340cc-kube-api-access-gl72m\") pod \"61be52ca-515b-4302-baac-f9350a9340cc\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") "
Sep 17 19:25:50.015457 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.015433    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61be52ca-515b-4302-baac-f9350a9340cc-kserve-provision-location\") pod \"61be52ca-515b-4302-baac-f9350a9340cc\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") "
Sep 17 19:25:50.015693 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.015465    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61be52ca-515b-4302-baac-f9350a9340cc-proxy-tls\") pod \"61be52ca-515b-4302-baac-f9350a9340cc\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") "
Sep 17 19:25:50.015693 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.015490    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-kube-rbac-proxy-sar-config\") pod \"61be52ca-515b-4302-baac-f9350a9340cc\" (UID: \"61be52ca-515b-4302-baac-f9350a9340cc\") "
Sep 17 19:25:50.015873 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.015843    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61be52ca-515b-4302-baac-f9350a9340cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61be52ca-515b-4302-baac-f9350a9340cc" (UID: "61be52ca-515b-4302-baac-f9350a9340cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:25:50.015873 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.015777    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "61be52ca-515b-4302-baac-f9350a9340cc" (UID: "61be52ca-515b-4302-baac-f9350a9340cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:25:50.016014 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.015905    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "61be52ca-515b-4302-baac-f9350a9340cc" (UID: "61be52ca-515b-4302-baac-f9350a9340cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:25:50.017709 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.017684    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61be52ca-515b-4302-baac-f9350a9340cc-kube-api-access-gl72m" (OuterVolumeSpecName: "kube-api-access-gl72m") pod "61be52ca-515b-4302-baac-f9350a9340cc" (UID: "61be52ca-515b-4302-baac-f9350a9340cc"). InnerVolumeSpecName "kube-api-access-gl72m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:25:50.017709 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.017697    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61be52ca-515b-4302-baac-f9350a9340cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61be52ca-515b-4302-baac-f9350a9340cc" (UID: "61be52ca-515b-4302-baac-f9350a9340cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:25:50.116995 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.116959    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-cabundle-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:25:50.116995 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.116990    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl72m\" (UniqueName: \"kubernetes.io/projected/61be52ca-515b-4302-baac-f9350a9340cc-kube-api-access-gl72m\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:25:50.116995 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.116999    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61be52ca-515b-4302-baac-f9350a9340cc-kserve-provision-location\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:25:50.117267 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.117009    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61be52ca-515b-4302-baac-f9350a9340cc-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:25:50.117267 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.117019    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61be52ca-515b-4302-baac-f9350a9340cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:25:50.286160 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.286060    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" event={"ID":"306f8a7c-69a7-483e-91ad-3d5295d5b514","Type":"ContainerStarted","Data":"48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494"}
Sep 17 19:25:50.286160 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.286133    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" event={"ID":"306f8a7c-69a7-483e-91ad-3d5295d5b514","Type":"ContainerStarted","Data":"3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc"}
Sep 17 19:25:50.286643 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.286426    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:50.286643 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.286468    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:50.287902 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.287871    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:25:50.288063 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.288042    2567 generic.go:356] "Generic (PLEG): container finished" podID="61be52ca-515b-4302-baac-f9350a9340cc" containerID="3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329" exitCode=0
Sep 17 19:25:50.288122 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.288102    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"
Sep 17 19:25:50.288193 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.288116    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" event={"ID":"61be52ca-515b-4302-baac-f9350a9340cc","Type":"ContainerDied","Data":"3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329"}
Sep 17 19:25:50.288193 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.288190    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn" event={"ID":"61be52ca-515b-4302-baac-f9350a9340cc","Type":"ContainerDied","Data":"bc9cdc63eb762aae4d3d278b0876b075c3b57856ea0c8dda65870e9ca235bae4"}
Sep 17 19:25:50.288258 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.288222    2567 scope.go:117] "RemoveContainer" containerID="60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5"
Sep 17 19:25:50.297078 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.297061    2567 scope.go:117] "RemoveContainer" containerID="3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329"
Sep 17 19:25:50.305424 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.305404    2567 scope.go:117] "RemoveContainer" containerID="74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c"
Sep 17 19:25:50.305565 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.305516    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podStartSLOduration=3.162124551 podStartE2EDuration="7.305498463s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.143373912s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:25:50.304957462 +0000 UTC m=+1173.231175886" watchObservedRunningTime="2026-09-17 19:25:50.305498463 +0000 UTC m=+1173.231716883"
Sep 17 19:25:50.313802 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.313782    2567 scope.go:117] "RemoveContainer" containerID="60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5"
Sep 17 19:25:50.314110 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:25:50.314087    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5\": container with ID starting with 60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5 not found: ID does not exist" containerID="60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5"
Sep 17 19:25:50.314259 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.314119    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5"} err="failed to get container status \"60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5\": rpc error: code = NotFound desc = could not find container \"60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5\": container with ID starting with 60e9c761ffb4c53830e7457a6fab3cb151bfc04d6fd990a102bdce046c2ae4a5 not found: ID does not exist"
Sep 17 19:25:50.314259 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.314152    2567 scope.go:117] "RemoveContainer" containerID="3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329"
Sep 17 19:25:50.314458 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:25:50.314441    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329\": container with ID starting with 3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329 not found: ID does not exist" containerID="3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329"
Sep 17 19:25:50.314499 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.314464    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329"} err="failed to get container status \"3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329\": rpc error: code = NotFound desc = could not find container \"3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329\": container with ID starting with 3adff027b86cf8202e46c0ba9d7506eaca5f22eafe85a13a6734aa94e0eb7329 not found: ID does not exist"
Sep 17 19:25:50.314499 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.314479    2567 scope.go:117] "RemoveContainer" containerID="74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c"
Sep 17 19:25:50.314704 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:25:50.314690    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c\": container with ID starting with 74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c not found: ID does not exist" containerID="74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c"
Sep 17 19:25:50.314747 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.314706    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c"} err="failed to get container status \"74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c\": rpc error: code = NotFound desc = could not find container \"74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c\": container with ID starting with 74d69b13b1a3111db371c2e93e49ac16f7a6b9e6894a9f1a4d699c283d5b319c not found: ID does not exist"
Sep 17 19:25:50.318593 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.318564    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"]
Sep 17 19:25:50.320537 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:50.320513    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bdd76-predictor-59549fccf4-gpjdn"]
Sep 17 19:25:51.083031 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:51.082987    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61be52ca-515b-4302-baac-f9350a9340cc" path="/var/lib/kubelet/pods/61be52ca-515b-4302-baac-f9350a9340cc/volumes"
Sep 17 19:25:51.294581 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:51.294543    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:25:56.299729 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:56.299698    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:25:56.300391 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:25:56.300364    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:26:00.784733 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:26:00.784697    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:06.300963 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:26:06.300925    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:26:16.301062 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:26:16.301013    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:26:19.071796 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:26:19.071762    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:26:19.072257 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:26:19.071762    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:26:19.076649 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:26:19.076624    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:26:19.076817 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:26:19.076625    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:26:26.301071 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:26:26.301037    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:26:36.300388 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:26:36.300341    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:26:46.300639 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:26:46.300603    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:26:56.300365 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:26:56.300316    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:27:01.078731 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:01.078678    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:27:06.898760 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:06.898714    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"]
Sep 17 19:27:06.901386 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:06.899014    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" containerID="cri-o://3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc" gracePeriod=30
Sep 17 19:27:06.901386 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:06.899054    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kube-rbac-proxy" containerID="cri-o://48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494" gracePeriod=30
Sep 17 19:27:06.901386 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:06.899221    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:27:07.618045 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:07.618009    2567 generic.go:356] "Generic (PLEG): container finished" podID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerID="48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494" exitCode=2
Sep 17 19:27:07.618346 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:07.618092    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" event={"ID":"306f8a7c-69a7-483e-91ad-3d5295d5b514","Type":"ContainerDied","Data":"48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494"}
Sep 17 19:27:11.295681 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:11.295640    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 19:27:13.839016 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:13.838982    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:15.232646 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:15.232604    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:16.295043 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:16.295000    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 19:27:16.900497 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:16.900458    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:27:18.901679 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:18.901648    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:18.981799 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:18.981759    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:19.038888 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.038853    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"]
Sep 17 19:27:19.039291 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.039261    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container"
Sep 17 19:27:19.039291 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.039293    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container"
Sep 17 19:27:19.039450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.039309    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="storage-initializer"
Sep 17 19:27:19.039450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.039318    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="storage-initializer"
Sep 17 19:27:19.039450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.039356    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kube-rbac-proxy"
Sep 17 19:27:19.039450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.039365    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kube-rbac-proxy"
Sep 17 19:27:19.039450 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.039440    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kserve-container"
Sep 17 19:27:19.039700 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.039457    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="61be52ca-515b-4302-baac-f9350a9340cc" containerName="kube-rbac-proxy"
Sep 17 19:27:19.042563 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.042542    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.045174 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.045151    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 19:27:19.045174 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.045166    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 19:27:19.053195 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.053169    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"]
Sep 17 19:27:19.175166 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.175052    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.175166 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.175093    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca915107-c9a7-4f29-9d4d-24f6a26b7596-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.175410 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.175201    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkmdl\" (UniqueName: \"kubernetes.io/projected/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-api-access-gkmdl\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.175410 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.175248    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.175410 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.175313    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.223392 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.223350    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:19.276692 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.276654    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca915107-c9a7-4f29-9d4d-24f6a26b7596-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.276892 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.276698    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkmdl\" (UniqueName: \"kubernetes.io/projected/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-api-access-gkmdl\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.276892 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.276727    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.276892 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.276758    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.276892 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.276810    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.277328 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.277297    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.277610 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.277593    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.277654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.277624    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.279466 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.279441    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca915107-c9a7-4f29-9d4d-24f6a26b7596-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.287488 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.287462    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkmdl\" (UniqueName: \"kubernetes.io/projected/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-api-access-gkmdl\") pod \"isvc-canary-promote-predictor-78b4c78fb9-wk2mc\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.353402 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.353367    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:19.489591 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.489494    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:19.633224 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.633190    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"]
Sep 17 19:27:19.635750 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:27:19.635721    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca915107_c9a7_4f29_9d4d_24f6a26b7596.slice/crio-a813151e60e6fc263f4db0a4330e4f5b6348b607cd1c2c5a11066eecdb5d5585 WatchSource:0}: Error finding container a813151e60e6fc263f4db0a4330e4f5b6348b607cd1c2c5a11066eecdb5d5585: Status 404 returned error can't find the container with id a813151e60e6fc263f4db0a4330e4f5b6348b607cd1c2c5a11066eecdb5d5585
Sep 17 19:27:19.669447 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:19.669417    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" event={"ID":"ca915107-c9a7-4f29-9d4d-24f6a26b7596","Type":"ContainerStarted","Data":"a813151e60e6fc263f4db0a4330e4f5b6348b607cd1c2c5a11066eecdb5d5585"}
Sep 17 19:27:20.674984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:20.674946    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" event={"ID":"ca915107-c9a7-4f29-9d4d-24f6a26b7596","Type":"ContainerStarted","Data":"dad76d663146bfb96f47f236d6e1814d46e6465fbdad012cac7e0472ecbbb7a3"}
Sep 17 19:27:21.295736 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:21.295696    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 19:27:21.295956 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:21.295841    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:27:24.693149 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:24.693112    2567 generic.go:356] "Generic (PLEG): container finished" podID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerID="dad76d663146bfb96f47f236d6e1814d46e6465fbdad012cac7e0472ecbbb7a3" exitCode=0
Sep 17 19:27:24.693572 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:24.693181    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" event={"ID":"ca915107-c9a7-4f29-9d4d-24f6a26b7596","Type":"ContainerDied","Data":"dad76d663146bfb96f47f236d6e1814d46e6465fbdad012cac7e0472ecbbb7a3"}
Sep 17 19:27:25.699690 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:25.699653    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" event={"ID":"ca915107-c9a7-4f29-9d4d-24f6a26b7596","Type":"ContainerStarted","Data":"c2e10de311b49f7fb2e6a470a2da93aaa6387c60d86d3216f9bfba7d721d900f"}
Sep 17 19:27:25.700178 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:25.699715    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" event={"ID":"ca915107-c9a7-4f29-9d4d-24f6a26b7596","Type":"ContainerStarted","Data":"4a38cbd45bf6fbd58593dbdd2fea86438cef1cc122f02e61bc40f3bd138562cf"}
Sep 17 19:27:25.700178 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:25.699917    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:25.720262 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:25.720203    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podStartSLOduration=2.670089264 podStartE2EDuration="6.720185649s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.050096385s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:27:25.719279678 +0000 UTC m=+1268.645498125" watchObservedRunningTime="2026-09-17 19:27:25.720185649 +0000 UTC m=+1268.646404071"
Sep 17 19:27:26.295244 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:26.295190    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 19:27:26.704671 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:26.704638    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:26.706064 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:26.706039    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:27:27.708759 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:27.708712    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:27:31.295736 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:31.295688    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 19:27:32.712990 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:32.712952    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:27:32.713634 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:32.713603    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:27:36.294872 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:36.294822    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 19:27:37.546605 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.546575    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:27:37.742542 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.742447    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/306f8a7c-69a7-483e-91ad-3d5295d5b514-kserve-provision-location\") pod \"306f8a7c-69a7-483e-91ad-3d5295d5b514\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") "
Sep 17 19:27:37.742542 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.742484    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfwxt\" (UniqueName: \"kubernetes.io/projected/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-api-access-rfwxt\") pod \"306f8a7c-69a7-483e-91ad-3d5295d5b514\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") "
Sep 17 19:27:37.742542 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.742507    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/306f8a7c-69a7-483e-91ad-3d5295d5b514-proxy-tls\") pod \"306f8a7c-69a7-483e-91ad-3d5295d5b514\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") "
Sep 17 19:27:37.742542 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.742525    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-cabundle-cert\") pod \"306f8a7c-69a7-483e-91ad-3d5295d5b514\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") "
Sep 17 19:27:37.742888 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.742584    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-rbac-proxy-sar-config\") pod \"306f8a7c-69a7-483e-91ad-3d5295d5b514\" (UID: \"306f8a7c-69a7-483e-91ad-3d5295d5b514\") "
Sep 17 19:27:37.742888 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.742807    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306f8a7c-69a7-483e-91ad-3d5295d5b514-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "306f8a7c-69a7-483e-91ad-3d5295d5b514" (UID: "306f8a7c-69a7-483e-91ad-3d5295d5b514"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:27:37.743009 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.742905    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "306f8a7c-69a7-483e-91ad-3d5295d5b514" (UID: "306f8a7c-69a7-483e-91ad-3d5295d5b514"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:37.743009 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.742963    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "306f8a7c-69a7-483e-91ad-3d5295d5b514" (UID: "306f8a7c-69a7-483e-91ad-3d5295d5b514"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:27:37.744755 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.744732    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306f8a7c-69a7-483e-91ad-3d5295d5b514-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "306f8a7c-69a7-483e-91ad-3d5295d5b514" (UID: "306f8a7c-69a7-483e-91ad-3d5295d5b514"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:27:37.744835 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.744788    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-api-access-rfwxt" (OuterVolumeSpecName: "kube-api-access-rfwxt") pod "306f8a7c-69a7-483e-91ad-3d5295d5b514" (UID: "306f8a7c-69a7-483e-91ad-3d5295d5b514"). InnerVolumeSpecName "kube-api-access-rfwxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:27:37.753665 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.753639    2567 generic.go:356] "Generic (PLEG): container finished" podID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerID="3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc" exitCode=137
Sep 17 19:27:37.753771 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.753697    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" event={"ID":"306f8a7c-69a7-483e-91ad-3d5295d5b514","Type":"ContainerDied","Data":"3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc"}
Sep 17 19:27:37.753771 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.753702    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"
Sep 17 19:27:37.753771 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.753747    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh" event={"ID":"306f8a7c-69a7-483e-91ad-3d5295d5b514","Type":"ContainerDied","Data":"4dcf760462486558acbd66d8cc4bb8ea52e9f6603ff2abea06ba75cf12ad4d30"}
Sep 17 19:27:37.753879 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.753775    2567 scope.go:117] "RemoveContainer" containerID="48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494"
Sep 17 19:27:37.763051 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.763031    2567 scope.go:117] "RemoveContainer" containerID="3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc"
Sep 17 19:27:37.770943 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.770917    2567 scope.go:117] "RemoveContainer" containerID="6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35"
Sep 17 19:27:37.777146 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.777109    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"]
Sep 17 19:27:37.779170 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.779151    2567 scope.go:117] "RemoveContainer" containerID="48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494"
Sep 17 19:27:37.779514 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:27:37.779492    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494\": container with ID starting with 48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494 not found: ID does not exist" containerID="48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494"
Sep 17 19:27:37.779575 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.779524    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494"} err="failed to get container status \"48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494\": rpc error: code = NotFound desc = could not find container \"48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494\": container with ID starting with 48f9bbb35647616bb33366e4ec87345e57f6247b96170081b9af1badd0b65494 not found: ID does not exist"
Sep 17 19:27:37.779575 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.779544    2567 scope.go:117] "RemoveContainer" containerID="3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc"
Sep 17 19:27:37.779803 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:27:37.779780    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc\": container with ID starting with 3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc not found: ID does not exist" containerID="3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc"
Sep 17 19:27:37.779863 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.779814    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc"} err="failed to get container status \"3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc\": rpc error: code = NotFound desc = could not find container \"3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc\": container with ID starting with 3e182674e9cf608dff16e511a305cc03111a0f63f71ae08f760cb86c16bbe5fc not found: ID does not exist"
Sep 17 19:27:37.779863 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.779837    2567 scope.go:117] "RemoveContainer" containerID="6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35"
Sep 17 19:27:37.780093 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:27:37.780072    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35\": container with ID starting with 6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35 not found: ID does not exist" containerID="6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35"
Sep 17 19:27:37.780206 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.780097    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35"} err="failed to get container status \"6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35\": rpc error: code = NotFound desc = could not find container \"6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35\": container with ID starting with 6f889b85cce45efe6f40ad2580ed8e8d48ef03967aed4bf4f97e0a0fddcc1b35 not found: ID does not exist"
Sep 17 19:27:37.783006 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.782984    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-wfcdh"]
Sep 17 19:27:37.843527 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.843489    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:27:37.843527 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.843520    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/306f8a7c-69a7-483e-91ad-3d5295d5b514-kserve-provision-location\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:27:37.843527 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.843532    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfwxt\" (UniqueName: \"kubernetes.io/projected/306f8a7c-69a7-483e-91ad-3d5295d5b514-kube-api-access-rfwxt\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:27:37.843815 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.843541    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/306f8a7c-69a7-483e-91ad-3d5295d5b514-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:27:37.843815 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:37.843552    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/306f8a7c-69a7-483e-91ad-3d5295d5b514-cabundle-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:27:39.084261 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:39.084222    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" path="/var/lib/kubelet/pods/306f8a7c-69a7-483e-91ad-3d5295d5b514/volumes"
Sep 17 19:27:42.714383 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:42.714341    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:27:52.713794 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:27:52.713755    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:28:02.714498 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:02.714451    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:28:12.714080 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:12.714037    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:28:22.714204 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:22.714160    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:28:32.713734 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:32.713648    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:28:41.898575 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:41.898540    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"]
Sep 17 19:28:41.899024 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:41.898779    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container" containerID="cri-o://4a38cbd45bf6fbd58593dbdd2fea86438cef1cc122f02e61bc40f3bd138562cf" gracePeriod=30
Sep 17 19:28:41.899024 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:41.898849    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kube-rbac-proxy" containerID="cri-o://c2e10de311b49f7fb2e6a470a2da93aaa6387c60d86d3216f9bfba7d721d900f" gracePeriod=30
Sep 17 19:28:41.899184 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:41.899033    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:28:42.024245 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:42.024214    2567 generic.go:356] "Generic (PLEG): container finished" podID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerID="c2e10de311b49f7fb2e6a470a2da93aaa6387c60d86d3216f9bfba7d721d900f" exitCode=2
Sep 17 19:28:42.024395 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:42.024282    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" event={"ID":"ca915107-c9a7-4f29-9d4d-24f6a26b7596","Type":"ContainerDied","Data":"c2e10de311b49f7fb2e6a470a2da93aaa6387c60d86d3216f9bfba7d721d900f"}
Sep 17 19:28:42.709473 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:42.709430    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 19:28:47.709658 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:47.709610    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 19:28:49.057953 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.057892    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:49.106775 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.106742    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"]
Sep 17 19:28:49.107177 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.107159    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kube-rbac-proxy"
Sep 17 19:28:49.107280 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.107179    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kube-rbac-proxy"
Sep 17 19:28:49.107280 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.107207    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="storage-initializer"
Sep 17 19:28:49.107280 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.107215    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="storage-initializer"
Sep 17 19:28:49.107280 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.107227    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container"
Sep 17 19:28:49.107280 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.107236    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container"
Sep 17 19:28:49.107531 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.107318    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kube-rbac-proxy"
Sep 17 19:28:49.107531 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.107335    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="306f8a7c-69a7-483e-91ad-3d5295d5b514" containerName="kserve-container"
Sep 17 19:28:49.110614 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.110591    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.113198 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.113177    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 19:28:49.120393 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.120367    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"]
Sep 17 19:28:49.146906 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.146867    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.147117 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.146937    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttz46\" (UniqueName: \"kubernetes.io/projected/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-api-access-ttz46\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.147117 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.146972    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.147117 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.147009    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.147117 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.147034    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.247926 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.247884    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttz46\" (UniqueName: \"kubernetes.io/projected/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-api-access-ttz46\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.247926 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.247931    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.248235 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.248054    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.248235 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.248089    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.248343 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.248258    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.248569 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.248546    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.248799 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.248780    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.248885 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.248867    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.250637 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.250621    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.257269 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.257242    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttz46\" (UniqueName: \"kubernetes.io/projected/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-api-access-ttz46\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.420678 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.420638    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:49.701239 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:49.701211    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"]
Sep 17 19:28:49.703677 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:28:49.703644    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c4d7c5_b393_4071_8b45_b03f5bd470dc.slice/crio-a537a33147805a1f0e1382ab8a4c3c4d472eeac34988e3f6e9feba9784601f55 WatchSource:0}: Error finding container a537a33147805a1f0e1382ab8a4c3c4d472eeac34988e3f6e9feba9784601f55: Status 404 returned error can't find the container with id a537a33147805a1f0e1382ab8a4c3c4d472eeac34988e3f6e9feba9784601f55
Sep 17 19:28:50.063701 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:50.063605    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" event={"ID":"d2c4d7c5-b393-4071-8b45-b03f5bd470dc","Type":"ContainerStarted","Data":"8f2035b636c92b4ea1aafeb8ce75972aa1d09c780835388d960dc59d7d7286d2"}
Sep 17 19:28:50.063701 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:50.063666    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" event={"ID":"d2c4d7c5-b393-4071-8b45-b03f5bd470dc","Type":"ContainerStarted","Data":"a537a33147805a1f0e1382ab8a4c3c4d472eeac34988e3f6e9feba9784601f55"}
Sep 17 19:28:51.900897 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:51.900861    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:28:52.708967 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:52.708927    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 19:28:52.709153 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:52.709046    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:28:54.082409 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:54.082374    2567 generic.go:356] "Generic (PLEG): container finished" podID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerID="8f2035b636c92b4ea1aafeb8ce75972aa1d09c780835388d960dc59d7d7286d2" exitCode=0
Sep 17 19:28:54.082786 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:54.082447    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" event={"ID":"d2c4d7c5-b393-4071-8b45-b03f5bd470dc","Type":"ContainerDied","Data":"8f2035b636c92b4ea1aafeb8ce75972aa1d09c780835388d960dc59d7d7286d2"}
Sep 17 19:28:55.088183 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:55.088126    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" event={"ID":"d2c4d7c5-b393-4071-8b45-b03f5bd470dc","Type":"ContainerStarted","Data":"0039728c2d07d56a3e3fc47076c0dc96c494ec80021ac12af5371b0653759ff5"}
Sep 17 19:28:55.088654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:55.088211    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" event={"ID":"d2c4d7c5-b393-4071-8b45-b03f5bd470dc","Type":"ContainerStarted","Data":"9e2c60710727c07dc6856cf7887481dc7561d2056a4a4f009fb669a97b9f59ef"}
Sep 17 19:28:55.088654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:55.088473    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:55.088654 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:55.088617    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:28:55.089949 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:55.089922    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:28:55.109055 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:55.109006    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podStartSLOduration=1.990512169 podStartE2EDuration="6.108991971s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.118479802s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:28:55.107718286 +0000 UTC m=+1358.033936710" watchObservedRunningTime="2026-09-17 19:28:55.108991971 +0000 UTC m=+1358.035210394"
Sep 17 19:28:56.093133 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:56.093083    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:28:57.709280 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:28:57.709232    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 19:29:01.097781 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:01.097752    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:29:01.098395 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:01.098365    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:29:02.709200 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:02.709130    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 19:29:07.709287 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:07.709245    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 19:29:11.099022 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:11.098963    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:29:12.161373 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.161276    2567 generic.go:356] "Generic (PLEG): container finished" podID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerID="4a38cbd45bf6fbd58593dbdd2fea86438cef1cc122f02e61bc40f3bd138562cf" exitCode=137
Sep 17 19:29:12.161373 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.161341    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" event={"ID":"ca915107-c9a7-4f29-9d4d-24f6a26b7596","Type":"ContainerDied","Data":"4a38cbd45bf6fbd58593dbdd2fea86438cef1cc122f02e61bc40f3bd138562cf"}
Sep 17 19:29:12.261745 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.261717    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:29:12.357061 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.357019    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca915107-c9a7-4f29-9d4d-24f6a26b7596-proxy-tls\") pod \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") "
Sep 17 19:29:12.357297 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.357077    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkmdl\" (UniqueName: \"kubernetes.io/projected/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-api-access-gkmdl\") pod \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") "
Sep 17 19:29:12.357297 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.357110    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-cabundle-cert\") pod \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") "
Sep 17 19:29:12.357297 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.357172    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-rbac-proxy-sar-config\") pod \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") "
Sep 17 19:29:12.357476 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.357415    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kserve-provision-location\") pod \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\" (UID: \"ca915107-c9a7-4f29-9d4d-24f6a26b7596\") "
Sep 17 19:29:12.357576 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.357554    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca915107-c9a7-4f29-9d4d-24f6a26b7596" (UID: "ca915107-c9a7-4f29-9d4d-24f6a26b7596"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:29:12.357617 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.357570    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca915107-c9a7-4f29-9d4d-24f6a26b7596" (UID: "ca915107-c9a7-4f29-9d4d-24f6a26b7596"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:29:12.357761 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.357734    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca915107-c9a7-4f29-9d4d-24f6a26b7596" (UID: "ca915107-c9a7-4f29-9d4d-24f6a26b7596"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:29:12.357831 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.357818    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-cabundle-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:29:12.357877 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.357838    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:29:12.357877 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.357852    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kserve-provision-location\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:29:12.359392 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.359369    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca915107-c9a7-4f29-9d4d-24f6a26b7596-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca915107-c9a7-4f29-9d4d-24f6a26b7596" (UID: "ca915107-c9a7-4f29-9d4d-24f6a26b7596"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:29:12.359468 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.359406    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-api-access-gkmdl" (OuterVolumeSpecName: "kube-api-access-gkmdl") pod "ca915107-c9a7-4f29-9d4d-24f6a26b7596" (UID: "ca915107-c9a7-4f29-9d4d-24f6a26b7596"). InnerVolumeSpecName "kube-api-access-gkmdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:29:12.459177 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.459048    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca915107-c9a7-4f29-9d4d-24f6a26b7596-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:29:12.459177 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:12.459083    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkmdl\" (UniqueName: \"kubernetes.io/projected/ca915107-c9a7-4f29-9d4d-24f6a26b7596-kube-api-access-gkmdl\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:29:13.167561 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:13.167526    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc" event={"ID":"ca915107-c9a7-4f29-9d4d-24f6a26b7596","Type":"ContainerDied","Data":"a813151e60e6fc263f4db0a4330e4f5b6348b607cd1c2c5a11066eecdb5d5585"}
Sep 17 19:29:13.167975 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:13.167567    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"
Sep 17 19:29:13.167975 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:13.167594    2567 scope.go:117] "RemoveContainer" containerID="c2e10de311b49f7fb2e6a470a2da93aaa6387c60d86d3216f9bfba7d721d900f"
Sep 17 19:29:13.175880 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:13.175861    2567 scope.go:117] "RemoveContainer" containerID="4a38cbd45bf6fbd58593dbdd2fea86438cef1cc122f02e61bc40f3bd138562cf"
Sep 17 19:29:13.183635 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:13.183609    2567 scope.go:117] "RemoveContainer" containerID="dad76d663146bfb96f47f236d6e1814d46e6465fbdad012cac7e0472ecbbb7a3"
Sep 17 19:29:13.186608 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:13.186582    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"]
Sep 17 19:29:13.188566 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:13.188542    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-wk2mc"]
Sep 17 19:29:15.083774 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:15.083742    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" path="/var/lib/kubelet/pods/ca915107-c9a7-4f29-9d4d-24f6a26b7596/volumes"
Sep 17 19:29:21.098299 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:21.098261    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:29:31.099189 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:31.099127    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:29:41.098857 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:41.098812    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:29:51.098991 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:29:51.098950    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:30:01.098878 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:01.098839    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:30:04.393815 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:04.393771    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:11.099251 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:11.099211    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:30:17.082300 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:17.082271    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:30:19.239798 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:19.239763    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"]
Sep 17 19:30:19.240356 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:19.240051    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container" containerID="cri-o://9e2c60710727c07dc6856cf7887481dc7561d2056a4a4f009fb669a97b9f59ef" gracePeriod=30
Sep 17 19:30:19.240356 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:19.240162    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kube-rbac-proxy" containerID="cri-o://0039728c2d07d56a3e3fc47076c0dc96c494ec80021ac12af5371b0653759ff5" gracePeriod=30
Sep 17 19:30:19.268473 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:19.268444    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:19.439276 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:19.439235    2567 generic.go:356] "Generic (PLEG): container finished" podID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerID="0039728c2d07d56a3e3fc47076c0dc96c494ec80021ac12af5371b0653759ff5" exitCode=2
Sep 17 19:30:19.439454 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:19.439340    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" event={"ID":"d2c4d7c5-b393-4071-8b45-b03f5bd470dc","Type":"ContainerDied","Data":"0039728c2d07d56a3e3fc47076c0dc96c494ec80021ac12af5371b0653759ff5"}
Sep 17 19:30:21.093398 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:21.093358    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 19:30:24.462390 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.462361    2567 generic.go:356] "Generic (PLEG): container finished" podID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerID="9e2c60710727c07dc6856cf7887481dc7561d2056a4a4f009fb669a97b9f59ef" exitCode=0
Sep 17 19:30:24.462729 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.462431    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" event={"ID":"d2c4d7c5-b393-4071-8b45-b03f5bd470dc","Type":"ContainerDied","Data":"9e2c60710727c07dc6856cf7887481dc7561d2056a4a4f009fb669a97b9f59ef"}
Sep 17 19:30:24.596677 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.596650    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:30:24.758838 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.758737    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-cabundle-cert\") pod \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") "
Sep 17 19:30:24.758838 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.758778    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kserve-provision-location\") pod \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") "
Sep 17 19:30:24.758838 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.758798    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-proxy-tls\") pod \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") "
Sep 17 19:30:24.758838 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.758821    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttz46\" (UniqueName: \"kubernetes.io/projected/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-api-access-ttz46\") pod \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") "
Sep 17 19:30:24.759198 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.758874    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-rbac-proxy-sar-config\") pod \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\" (UID: \"d2c4d7c5-b393-4071-8b45-b03f5bd470dc\") "
Sep 17 19:30:24.759198 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.759159    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2c4d7c5-b393-4071-8b45-b03f5bd470dc" (UID: "d2c4d7c5-b393-4071-8b45-b03f5bd470dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:30:24.759299 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.759173    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2c4d7c5-b393-4071-8b45-b03f5bd470dc" (UID: "d2c4d7c5-b393-4071-8b45-b03f5bd470dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:24.759386 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.759363    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2c4d7c5-b393-4071-8b45-b03f5bd470dc" (UID: "d2c4d7c5-b393-4071-8b45-b03f5bd470dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:30:24.761193 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.761166    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-api-access-ttz46" (OuterVolumeSpecName: "kube-api-access-ttz46") pod "d2c4d7c5-b393-4071-8b45-b03f5bd470dc" (UID: "d2c4d7c5-b393-4071-8b45-b03f5bd470dc"). InnerVolumeSpecName "kube-api-access-ttz46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:30:24.761297 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.761251    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2c4d7c5-b393-4071-8b45-b03f5bd470dc" (UID: "d2c4d7c5-b393-4071-8b45-b03f5bd470dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:30:24.859660 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.859622    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:30:24.859660 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.859653    2567 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-cabundle-cert\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:30:24.859660 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.859666    2567 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kserve-provision-location\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:30:24.859904 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.859678    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:30:24.859904 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:24.859690    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttz46\" (UniqueName: \"kubernetes.io/projected/d2c4d7c5-b393-4071-8b45-b03f5bd470dc-kube-api-access-ttz46\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:30:25.468774 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:25.468740    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn" event={"ID":"d2c4d7c5-b393-4071-8b45-b03f5bd470dc","Type":"ContainerDied","Data":"a537a33147805a1f0e1382ab8a4c3c4d472eeac34988e3f6e9feba9784601f55"}
Sep 17 19:30:25.469244 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:25.468807    2567 scope.go:117] "RemoveContainer" containerID="0039728c2d07d56a3e3fc47076c0dc96c494ec80021ac12af5371b0653759ff5"
Sep 17 19:30:25.469244 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:25.468758    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"
Sep 17 19:30:25.477172 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:25.477131    2567 scope.go:117] "RemoveContainer" containerID="9e2c60710727c07dc6856cf7887481dc7561d2056a4a4f009fb669a97b9f59ef"
Sep 17 19:30:25.484455 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:25.484431    2567 scope.go:117] "RemoveContainer" containerID="8f2035b636c92b4ea1aafeb8ce75972aa1d09c780835388d960dc59d7d7286d2"
Sep 17 19:30:25.488519 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:25.488494    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"]
Sep 17 19:30:25.494809 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:25.494788    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-g9bkn"]
Sep 17 19:30:27.082120 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:27.082085    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" path="/var/lib/kubelet/pods/d2c4d7c5-b393-4071-8b45-b03f5bd470dc/volumes"
Sep 17 19:30:33.740317 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.740275    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:33.775941 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.775904    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"]
Sep 17 19:30:33.776275 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776262    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container"
Sep 17 19:30:33.776324 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776277    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container"
Sep 17 19:30:33.776324 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776288    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="storage-initializer"
Sep 17 19:30:33.776324 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776294    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="storage-initializer"
Sep 17 19:30:33.776324 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776301    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="storage-initializer"
Sep 17 19:30:33.776324 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776307    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="storage-initializer"
Sep 17 19:30:33.776324 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776315    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container"
Sep 17 19:30:33.776324 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776321    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container"
Sep 17 19:30:33.776534 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776335    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kube-rbac-proxy"
Sep 17 19:30:33.776534 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776341    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kube-rbac-proxy"
Sep 17 19:30:33.776534 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776352    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kube-rbac-proxy"
Sep 17 19:30:33.776534 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776357    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kube-rbac-proxy"
Sep 17 19:30:33.776534 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776403    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kube-rbac-proxy"
Sep 17 19:30:33.776534 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776410    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca915107-c9a7-4f29-9d4d-24f6a26b7596" containerName="kserve-container"
Sep 17 19:30:33.776534 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776417    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kserve-container"
Sep 17 19:30:33.776534 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.776423    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c4d7c5-b393-4071-8b45-b03f5bd470dc" containerName="kube-rbac-proxy"
Sep 17 19:30:33.780902 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.780883    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:33.783723 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.783702    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vpx7x\""
Sep 17 19:30:33.783857 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.783704    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 19:30:33.783857 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.783702    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 17 19:30:33.785429 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.785398    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:30:33.785429 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.785398    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:30:33.789696 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.789674    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"]
Sep 17 19:30:33.938720 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.938658    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043375a7-e9e4-4f5e-a657-9433b64973f8-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bbbch\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:33.938720 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.938722    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bbbch\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:33.938958 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:33.938801    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n82xh\" (UniqueName: \"kubernetes.io/projected/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-api-access-n82xh\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bbbch\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:34.040063 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.039971    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043375a7-e9e4-4f5e-a657-9433b64973f8-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bbbch\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:34.040063 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.040017    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bbbch\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:34.040063 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.040055    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n82xh\" (UniqueName: \"kubernetes.io/projected/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-api-access-n82xh\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bbbch\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:34.040819 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.040790    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bbbch\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:34.042683 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.042648    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043375a7-e9e4-4f5e-a657-9433b64973f8-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bbbch\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:34.049756 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.049724    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n82xh\" (UniqueName: \"kubernetes.io/projected/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-api-access-n82xh\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-bbbch\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:34.065049 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.065019    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:34.091195 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.091167    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:34.363743 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.363712    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"]
Sep 17 19:30:34.367347 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:30:34.367319    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043375a7_e9e4_4f5e_a657_9433b64973f8.slice/crio-18e708396aaa55d83ce7dbc077078c6486b2e0493ac8045838a263b4a63ee773 WatchSource:0}: Error finding container 18e708396aaa55d83ce7dbc077078c6486b2e0493ac8045838a263b4a63ee773: Status 404 returned error can't find the container with id 18e708396aaa55d83ce7dbc077078c6486b2e0493ac8045838a263b4a63ee773
Sep 17 19:30:34.507537 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.507501    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch" event={"ID":"043375a7-e9e4-4f5e-a657-9433b64973f8","Type":"ContainerStarted","Data":"bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3"}
Sep 17 19:30:34.507728 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.507567    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:34.507728 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.507590    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:34.507728 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.507606    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch" event={"ID":"043375a7-e9e4-4f5e-a657-9433b64973f8","Type":"ContainerStarted","Data":"1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a"}
Sep 17 19:30:34.507728 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.507623    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch" event={"ID":"043375a7-e9e4-4f5e-a657-9433b64973f8","Type":"ContainerStarted","Data":"18e708396aaa55d83ce7dbc077078c6486b2e0493ac8045838a263b4a63ee773"}
Sep 17 19:30:34.509530 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.509511    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:34.530423 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:34.530372    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch" podStartSLOduration=1.5303489300000002 podStartE2EDuration="1.53034893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:30:34.528699424 +0000 UTC m=+1457.454917857" watchObservedRunningTime="2026-09-17 19:30:34.53034893 +0000 UTC m=+1457.456567352"
Sep 17 19:30:41.521560 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:41.521531    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:30:43.771602 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:43.771571    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:44.017783 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:30:44.017746    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:31:19.098641 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:31:19.098609    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:31:19.100733 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:31:19.100707    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:31:19.103103 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:31:19.103081    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:31:19.105004 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:31:19.104985    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:32:25.140837 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:25.140801    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:29.285435 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.285404    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-bbbch_043375a7-e9e4-4f5e-a657-9433b64973f8/kserve-container/0.log"
Sep 17 19:32:29.646180 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.646120    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:29.659862 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.659820    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"]
Sep 17 19:32:29.660119 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.660094    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch" podUID="043375a7-e9e4-4f5e-a657-9433b64973f8" containerName="kserve-container" containerID="cri-o://1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a" gracePeriod=30
Sep 17 19:32:29.660228 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.660106    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch" podUID="043375a7-e9e4-4f5e-a657-9433b64973f8" containerName="kube-rbac-proxy" containerID="cri-o://bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3" gracePeriod=30
Sep 17 19:32:29.723475 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.723440    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:29.907396 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.907335    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:32:29.918769 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.918741    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043375a7-e9e4-4f5e-a657-9433b64973f8-proxy-tls\") pod \"043375a7-e9e4-4f5e-a657-9433b64973f8\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") "
Sep 17 19:32:29.918916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.918836    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n82xh\" (UniqueName: \"kubernetes.io/projected/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-api-access-n82xh\") pod \"043375a7-e9e4-4f5e-a657-9433b64973f8\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") "
Sep 17 19:32:29.918916 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.918872    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-rbac-proxy-sar-config\") pod \"043375a7-e9e4-4f5e-a657-9433b64973f8\" (UID: \"043375a7-e9e4-4f5e-a657-9433b64973f8\") "
Sep 17 19:32:29.919284 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.919257    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "043375a7-e9e4-4f5e-a657-9433b64973f8" (UID: "043375a7-e9e4-4f5e-a657-9433b64973f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:32:29.921000 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.920974    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/043375a7-e9e4-4f5e-a657-9433b64973f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "043375a7-e9e4-4f5e-a657-9433b64973f8" (UID: "043375a7-e9e4-4f5e-a657-9433b64973f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:32:29.921000 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.920993    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-api-access-n82xh" (OuterVolumeSpecName: "kube-api-access-n82xh") pod "043375a7-e9e4-4f5e-a657-9433b64973f8" (UID: "043375a7-e9e4-4f5e-a657-9433b64973f8"). InnerVolumeSpecName "kube-api-access-n82xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:32:29.981126 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.981080    2567 generic.go:356] "Generic (PLEG): container finished" podID="043375a7-e9e4-4f5e-a657-9433b64973f8" containerID="bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3" exitCode=2
Sep 17 19:32:29.981126 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.981109    2567 generic.go:356] "Generic (PLEG): container finished" podID="043375a7-e9e4-4f5e-a657-9433b64973f8" containerID="1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a" exitCode=2
Sep 17 19:32:29.981126 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.981148    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"
Sep 17 19:32:29.981412 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.981163    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch" event={"ID":"043375a7-e9e4-4f5e-a657-9433b64973f8","Type":"ContainerDied","Data":"bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3"}
Sep 17 19:32:29.981412 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.981240    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch" event={"ID":"043375a7-e9e4-4f5e-a657-9433b64973f8","Type":"ContainerDied","Data":"1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a"}
Sep 17 19:32:29.981412 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.981274    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch" event={"ID":"043375a7-e9e4-4f5e-a657-9433b64973f8","Type":"ContainerDied","Data":"18e708396aaa55d83ce7dbc077078c6486b2e0493ac8045838a263b4a63ee773"}
Sep 17 19:32:29.981412 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.981304    2567 scope.go:117] "RemoveContainer" containerID="bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3"
Sep 17 19:32:29.989984 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.989966    2567 scope.go:117] "RemoveContainer" containerID="1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a"
Sep 17 19:32:29.999712 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:29.999682    2567 scope.go:117] "RemoveContainer" containerID="bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3"
Sep 17 19:32:30.000177 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:32:30.000156    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3\": container with ID starting with bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3 not found: ID does not exist" containerID="bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3"
Sep 17 19:32:30.000235 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.000187    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3"} err="failed to get container status \"bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3\": rpc error: code = NotFound desc = could not find container \"bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3\": container with ID starting with bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3 not found: ID does not exist"
Sep 17 19:32:30.000235 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.000205    2567 scope.go:117] "RemoveContainer" containerID="1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a"
Sep 17 19:32:30.000524 ip-10-0-140-225 kubenswrapper[2567]: E0917 19:32:30.000490    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a\": container with ID starting with 1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a not found: ID does not exist" containerID="1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a"
Sep 17 19:32:30.000592 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.000533    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a"} err="failed to get container status \"1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a\": rpc error: code = NotFound desc = could not find container \"1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a\": container with ID starting with 1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a not found: ID does not exist"
Sep 17 19:32:30.000592 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.000555    2567 scope.go:117] "RemoveContainer" containerID="bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3"
Sep 17 19:32:30.000823 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.000805    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3"} err="failed to get container status \"bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3\": rpc error: code = NotFound desc = could not find container \"bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3\": container with ID starting with bd59fcd6027ab7a6a48304c5535baa3507140979676324b1f5798139237a2ee3 not found: ID does not exist"
Sep 17 19:32:30.000876 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.000833    2567 scope.go:117] "RemoveContainer" containerID="1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a"
Sep 17 19:32:30.001074 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.001058    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a"} err="failed to get container status \"1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a\": rpc error: code = NotFound desc = could not find container \"1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a\": container with ID starting with 1b7903a19686d3d9287b9d5c273c23a196c59c26b106fa916e3ffe721b12723a not found: ID does not exist"
Sep 17 19:32:30.009540 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.007751    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"]
Sep 17 19:32:30.012513 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.012487    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-bbbch"]
Sep 17 19:32:30.019519 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.019499    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n82xh\" (UniqueName: \"kubernetes.io/projected/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-api-access-n82xh\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:32:30.019583 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.019522    2567 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/043375a7-e9e4-4f5e-a657-9433b64973f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:32:30.019583 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.019534    2567 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/043375a7-e9e4-4f5e-a657-9433b64973f8-proxy-tls\") on node \"ip-10-0-140-225.ec2.internal\" DevicePath \"\""
Sep 17 19:32:30.327000 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.326885    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:30.396439 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:30.396406    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:31.083085 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:32:31.083049    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="043375a7-e9e4-4f5e-a657-9433b64973f8" path="/var/lib/kubelet/pods/043375a7-e9e4-4f5e-a657-9433b64973f8/volumes"
Sep 17 19:33:01.643066 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:01.642976    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:01.801103 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:01.801072    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:01.901158 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:01.901050    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:04.746510 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:04.746481    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wx589_4da49179-a694-49ea-a836-7ce5e0fbfb13/global-pull-secret-syncer/0.log"
Sep 17 19:33:04.798158 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:04.798121    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log"
Sep 17 19:33:04.940980 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:04.940950    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-225.ec2.internal_0937b68b17a7b35dfeea96e89aad3eb3/haproxy/0.log"
Sep 17 19:33:07.336946 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:07.336909    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:07.438536 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:07.438486    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2z92j_e6ebba4c-75f7-4b76-a077-c9bbbfd109e0/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:09.001121 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:09.001090    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-97zqg_c653f4cb-bc89-4470-9d85-f6550ff7f325/cluster-monitoring-operator/0.log"
Sep 17 19:33:09.130947 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:09.130923    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-676886d4d4-twqmc_ff385afc-848d-4ac4-85fd-a1c8c9629ee5/metrics-server/0.log"
Sep 17 19:33:09.196635 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:09.196601    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lm4j6_0de4d8a3-1226-4b73-a3d6-b52e3795f94e/node-exporter/0.log"
Sep 17 19:33:09.222149 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:09.222107    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lm4j6_0de4d8a3-1226-4b73-a3d6-b52e3795f94e/kube-rbac-proxy/0.log"
Sep 17 19:33:09.249240 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:09.249214    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lm4j6_0de4d8a3-1226-4b73-a3d6-b52e3795f94e/init-textfile/0.log"
Sep 17 19:33:09.855491 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:09.855459    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fff9667cb-kkgh4_ca4dada6-f4d0-4f5d-b958-25c7a0f659ac/telemeter-client/0.log"
Sep 17 19:33:09.882697 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:09.882670    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fff9667cb-kkgh4_ca4dada6-f4d0-4f5d-b958-25c7a0f659ac/reload/0.log"
Sep 17 19:33:09.916634 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:09.916606    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fff9667cb-kkgh4_ca4dada6-f4d0-4f5d-b958-25c7a0f659ac/kube-rbac-proxy/0.log"
Sep 17 19:33:09.967364 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:09.967338    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/thanos-query/0.log"
Sep 17 19:33:09.999447 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:09.999420    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/kube-rbac-proxy-web/0.log"
Sep 17 19:33:10.023942 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:10.023913    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/kube-rbac-proxy/0.log"
Sep 17 19:33:10.049085 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:10.049052    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/prom-label-proxy/0.log"
Sep 17 19:33:10.076012 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:10.075984    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/kube-rbac-proxy-rules/0.log"
Sep 17 19:33:10.102042 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:10.102019    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c78b66bb-wqrwb_83d76dee-ce50-4e39-a6a5-7e8e43c19fa2/kube-rbac-proxy-metrics/0.log"
Sep 17 19:33:12.091591 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:12.091561    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/2.log"
Sep 17 19:33:12.095239 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:12.095218    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-2ztxw_76e839db-c4d0-4f68-b9e6-3f8950193307/console-operator/3.log"
Sep 17 19:33:12.562489 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:12.562457    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-thnpp_0d51a30f-1aa8-4485-811c-2a7f0b5e9368/download-server/0.log"
Sep 17 19:33:13.223502 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.223467    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"]
Sep 17 19:33:13.223895 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.223772    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043375a7-e9e4-4f5e-a657-9433b64973f8" containerName="kserve-container"
Sep 17 19:33:13.223895 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.223783    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="043375a7-e9e4-4f5e-a657-9433b64973f8" containerName="kserve-container"
Sep 17 19:33:13.223895 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.223792    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043375a7-e9e4-4f5e-a657-9433b64973f8" containerName="kube-rbac-proxy"
Sep 17 19:33:13.223895 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.223798    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="043375a7-e9e4-4f5e-a657-9433b64973f8" containerName="kube-rbac-proxy"
Sep 17 19:33:13.223895 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.223854    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="043375a7-e9e4-4f5e-a657-9433b64973f8" containerName="kserve-container"
Sep 17 19:33:13.223895 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.223861    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="043375a7-e9e4-4f5e-a657-9433b64973f8" containerName="kube-rbac-proxy"
Sep 17 19:33:13.226688 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.226669    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.229497 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.229481    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n9b8n\"/\"default-dockercfg-qplkf\""
Sep 17 19:33:13.229568 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.229539    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n9b8n\"/\"openshift-service-ca.crt\""
Sep 17 19:33:13.230817 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.230794    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n9b8n\"/\"kube-root-ca.crt\""
Sep 17 19:33:13.236388 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.236366    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"]
Sep 17 19:33:13.379026 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.378991    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-sys\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.379254 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.379041    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwcv2\" (UniqueName: \"kubernetes.io/projected/07e9ec56-4378-454e-a3c3-a90657d72b9b-kube-api-access-dwcv2\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.379254 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.379061    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-proc\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.379254 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.379185    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-lib-modules\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.379254 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.379221    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-podres\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.479752 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.479659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwcv2\" (UniqueName: \"kubernetes.io/projected/07e9ec56-4378-454e-a3c3-a90657d72b9b-kube-api-access-dwcv2\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.479752 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.479697    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-proc\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.479752 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.479736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-lib-modules\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.479993 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.479759    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-podres\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.479993 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.479792    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-sys\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.479993 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.479807    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-proc\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.479993 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.479862    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-sys\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.479993 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.479913    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-podres\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.479993 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.479935    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07e9ec56-4378-454e-a3c3-a90657d72b9b-lib-modules\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.488949 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.488923    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwcv2\" (UniqueName: \"kubernetes.io/projected/07e9ec56-4378-454e-a3c3-a90657d72b9b-kube-api-access-dwcv2\") pod \"perf-node-gather-daemonset-bwt89\" (UID: \"07e9ec56-4378-454e-a3c3-a90657d72b9b\") " pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.536779 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.536735    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:13.809312 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:13.809131    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"]
Sep 17 19:33:13.812178 ip-10-0-140-225 kubenswrapper[2567]: W0917 19:33:13.812132    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod07e9ec56_4378_454e_a3c3_a90657d72b9b.slice/crio-d5f4f5ec986843e7975671e572a82144f225fc48138cdb30b8e49e31247b7f6d WatchSource:0}: Error finding container d5f4f5ec986843e7975671e572a82144f225fc48138cdb30b8e49e31247b7f6d: Status 404 returned error can't find the container with id d5f4f5ec986843e7975671e572a82144f225fc48138cdb30b8e49e31247b7f6d
Sep 17 19:33:14.060479 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:14.060376    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sj5qz_23034e81-6d79-4221-a617-c31e27bfa62e/dns/0.log"
Sep 17 19:33:14.083006 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:14.082977    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sj5qz_23034e81-6d79-4221-a617-c31e27bfa62e/kube-rbac-proxy/0.log"
Sep 17 19:33:14.158302 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:14.158259    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89" event={"ID":"07e9ec56-4378-454e-a3c3-a90657d72b9b","Type":"ContainerStarted","Data":"b0c2dcd2eb0784e83298d6dd77d7d1ea250673948a6f4b91918a6c56b55c6371"}
Sep 17 19:33:14.158495 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:14.158322    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89" event={"ID":"07e9ec56-4378-454e-a3c3-a90657d72b9b","Type":"ContainerStarted","Data":"d5f4f5ec986843e7975671e572a82144f225fc48138cdb30b8e49e31247b7f6d"}
Sep 17 19:33:14.158495 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:14.158350    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:14.175947 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:14.175892    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89" podStartSLOduration=1.175874226 podStartE2EDuration="1.175874226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:33:14.173356568 +0000 UTC m=+1617.099574992" watchObservedRunningTime="2026-09-17 19:33:14.175874226 +0000 UTC m=+1617.102092648"
Sep 17 19:33:14.206887 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:14.206857    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xhmnn_5744cdd7-665f-41f2-96b7-150f693100a2/dns-node-resolver/0.log"
Sep 17 19:33:14.691287 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:14.691252    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-557d646d44-6wm5d_c235293a-4120-48f2-8a31-21b5209b67a0/registry/0.log"
Sep 17 19:33:14.738057 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:14.738026    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g5mvq_f8c8d54f-7756-4625-94b7-806ff2e8db82/node-ca/0.log"
Sep 17 19:33:15.523256 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:15.523227    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-55b68b7566-djg2w_5e99e64b-b688-47b4-9271-04fce9af21f3/router/0.log"
Sep 17 19:33:15.905366 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:15.905339    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wlgn2_77e6ea87-9a6c-4157-83c6-fc3e356604d2/serve-healthcheck-canary/0.log"
Sep 17 19:33:16.250348 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:16.250259    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-kj7p7_da661868-318a-4f63-8068-f57f60e3f5e7/insights-operator/0.log"
Sep 17 19:33:16.252003 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:16.251979    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-kj7p7_da661868-318a-4f63-8068-f57f60e3f5e7/insights-operator/1.log"
Sep 17 19:33:16.270974 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:16.270943    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bljts_0724dd7f-24c9-4d71-97da-76bb064bd722/kube-rbac-proxy/0.log"
Sep 17 19:33:16.291159 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:16.291123    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bljts_0724dd7f-24c9-4d71-97da-76bb064bd722/exporter/0.log"
Sep 17 19:33:16.312079 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:16.312034    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bljts_0724dd7f-24c9-4d71-97da-76bb064bd722/extractor/0.log"
Sep 17 19:33:18.772212 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:18.772183    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-9bzlb_b491aad8-785b-4e38-b7d3-d4bafdb77743/manager/0.log"
Sep 17 19:33:20.172541 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:20.172514    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n9b8n/perf-node-gather-daemonset-bwt89"
Sep 17 19:33:23.532027 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:23.531939    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-kjzn7_ccc026e0-deda-423a-bd1d-22e29325633d/migrator/0.log"
Sep 17 19:33:23.556443 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:23.556412    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-kjzn7_ccc026e0-deda-423a-bd1d-22e29325633d/graceful-termination/0.log"
Sep 17 19:33:24.914708 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:24.914628    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwmhb_c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f/kube-multus-additional-cni-plugins/0.log"
Sep 17 19:33:24.937005 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:24.936982    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwmhb_c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f/egress-router-binary-copy/0.log"
Sep 17 19:33:24.960123 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:24.960097    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwmhb_c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f/cni-plugins/0.log"
Sep 17 19:33:24.985699 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:24.985674    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwmhb_c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f/bond-cni-plugin/0.log"
Sep 17 19:33:25.013647 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:25.013619    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwmhb_c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f/routeoverride-cni/0.log"
Sep 17 19:33:25.036514 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:25.036488    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwmhb_c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f/whereabouts-cni-bincopy/0.log"
Sep 17 19:33:25.062030 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:25.062003    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fwmhb_c6a8f197-4b3c-4ddc-8a7e-00d2e0a2e80f/whereabouts-cni/0.log"
Sep 17 19:33:25.439291 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:25.439262    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-c5cgz_a31cbf4c-8c80-4d28-b1b4-4bb27aedb301/kube-multus/0.log"
Sep 17 19:33:25.497593 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:25.497560    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8gl6r_223bc150-3bae-4e3a-a45e-69865f53a23d/network-metrics-daemon/0.log"
Sep 17 19:33:25.518253 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:25.518221    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8gl6r_223bc150-3bae-4e3a-a45e-69865f53a23d/kube-rbac-proxy/0.log"
Sep 17 19:33:26.366812 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:26.366787    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-controller/0.log"
Sep 17 19:33:26.386086 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:26.386060    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/1.log"
Sep 17 19:33:26.386271 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:26.386095    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovn-acl-logging/0.log"
Sep 17 19:33:26.405099 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:26.405070    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/kube-rbac-proxy-node/0.log"
Sep 17 19:33:26.432631 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:26.432601    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 19:33:26.455476 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:26.455445    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/northd/0.log"
Sep 17 19:33:26.476574 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:26.476544    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/nbdb/0.log"
Sep 17 19:33:26.497359 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:26.497334    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/sbdb/0.log"
Sep 17 19:33:26.540874 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:26.540846    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cw78b_ea298d9d-0b60-4b56-92e5-2b258c2b5bff/ovnkube-controller/0.log"
Sep 17 19:33:28.214709 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:28.214682    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-mshnf_f663602d-7308-4423-880f-327ce634f45b/network-check-target-container/0.log"
Sep 17 19:33:29.201623 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:29.201596    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-7j5gq_fbadfc18-65c1-4256-b7da-f153a1a188e9/iptables-alerter/0.log"
Sep 17 19:33:29.959612 ip-10-0-140-225 kubenswrapper[2567]: I0917 19:33:29.959573    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-gvnfj_e8b40d82-e389-4df4-8ec9-dbf1b57f54a6/tuned/0.log"