Sep 19 08:20:45.883635 ip-10-0-134-26 systemd[1]: Starting Kubernetes Kubelet...
Sep 19 08:20:49.714224 ip-10-0-134-26 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:49.714224 ip-10-0-134-26 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 19 08:20:49.714224 ip-10-0-134-26 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:49.714224 ip-10-0-134-26 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 19 08:20:49.714224 ip-10-0-134-26 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:49.715144 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.714351    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 19 08:20:49.718193 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718174    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.718193 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718192    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718197    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718202    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718207    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718212    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718216    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718219    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718222    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718225    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718229    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718231    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718235    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718239    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718242    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718245    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718253    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718257    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718260    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718263    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718266    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.718261 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718269    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718272    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718275    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718278    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718281    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718285    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718287    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718290    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718293    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718296    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718299    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718301    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718304    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718307    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718310    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718312    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718315    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718318    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718320    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718323    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.718796 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718340    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718344    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718347    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718350    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718352    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718355    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718357    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718360    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718363    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718368    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718372    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718377    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718380    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718384    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718388    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718391    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718394    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718397    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718400    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.719346 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718403    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718406    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718409    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718411    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718414    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718417    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718420    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718423    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718426    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718428    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718431    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718435    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718437    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718440    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718443    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718446    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718450    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718453    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718456    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718458    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.719840 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718461    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718464    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718466    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718469    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718472    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718474    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718477    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718482    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718485    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718488    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718491    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718494    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718497    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718499    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718502    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718505    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718508    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718511    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.718514    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719266    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.720431 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719273    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719277    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719279    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719282    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719292    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719295    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719298    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719301    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719304    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719307    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719312    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719316    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719319    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719322    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719325    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719345    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719348    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719352    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719355    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719359    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.721015 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719362    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719366    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719369    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719371    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719374    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719377    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719380    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719383    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719386    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719388    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719391    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719394    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719397    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719400    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719403    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719405    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719408    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719413    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719415    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719418    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.721616 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719421    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719424    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719427    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719430    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719432    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719435    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719438    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719441    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719444    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719447    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719450    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719453    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719456    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719459    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719461    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719464    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719467    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719469    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719472    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719475    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.722146 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719478    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719480    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719486    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719489    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719493    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719496    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719498    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719501    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719505    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719507    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719511    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719513    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719516    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719519    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719522    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719525    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719527    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719530    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719533    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.722689 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719536    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719539    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719543    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719547    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719550    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719553    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719556    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719559    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719561    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719564    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719567    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719569    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719572    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719574    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719577    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719580    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719583    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719585    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.719588    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720054    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.723198 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720060    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720064    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720067    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720071    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720075    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720078    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720081    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720086    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720090    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720093    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720096    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720099    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720102    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720105    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720108    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720110    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720113    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720116    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720119    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720122    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.723755 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720125    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720127    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720130    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720133    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720136    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720139    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720141    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720144    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720147    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720149    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720152    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720157    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720159    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720162    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720166    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720169    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720178    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720181    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720184    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.724342 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720187    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720189    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720192    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720195    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720198    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720201    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720203    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720212    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720215    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720218    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720220    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720224    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720229    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720232    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720234    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720237    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720240    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720243    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720246    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720249    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.724845 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720251    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720254    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720257    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720259    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720264    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720266    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720269    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720272    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720275    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720277    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720286    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720289    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720292    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720294    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720297    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720300    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720302    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720305    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720309    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720311    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.725403 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720314    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720317    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720319    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720323    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720344    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720348    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720352    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720355    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720358    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720361    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720364    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720367    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720370    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720372    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720375    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720378    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720382    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720385    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.720388    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.725926 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720468    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720503    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720514    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720521    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720528    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720533    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720539    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720546    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720551    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720554    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720558    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720562    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720567    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720571    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720576    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720579    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720583    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720589    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720595    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720598    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720601    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720605    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720608    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720613    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 19 08:20:49.726449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720616    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720619    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720623    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720626    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720630    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720633    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720637    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720640    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720644    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720647    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720650    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720653    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720657    2559 flags.go:64] FLAG: --enable-server="true"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720660    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720664    2559 flags.go:64] FLAG: --event-burst="100"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720668    2559 flags.go:64] FLAG: --event-qps="50"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720671    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720674    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720677    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720681    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720684    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720687    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720690    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720693    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720696    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 19 08:20:49.727067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720700    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720703    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720706    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720710    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720713    2559 flags.go:64] FLAG: --feature-gates=""
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720717    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720720    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720724    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720727    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720730    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720734    2559 flags.go:64] FLAG: --help="false"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720737    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-134-26.ec2.internal"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720740    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720744    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720747    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720751    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720754    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720757    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720761    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720763    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720766    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720770    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720773    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720776    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 19 08:20:49.727809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720780    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720782    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720786    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720788    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720791    2559 flags.go:64] FLAG: --lock-file=""
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720794    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720797    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720800    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720807    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720810    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720813    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720816    2559 flags.go:64] FLAG: --logging-format="text"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720819    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720823    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720826    2559 flags.go:64] FLAG: --manifest-url=""
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720829    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720834    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720837    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720841    2559 flags.go:64] FLAG: --max-pods="110"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720844    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720848    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720851    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720854    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720857    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720861    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 19 08:20:49.728473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720864    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720871    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720875    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720878    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720881    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720886    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720892    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720895    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720898    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720902    2559 flags.go:64] FLAG: --port="10250"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720905    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720908    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05d5e736edeb8818c"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720911    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720915    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720918    2559 flags.go:64] FLAG: --register-node="true"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720921    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720925    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720929    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720932    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720935    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720939    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720942    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720945    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720949    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720952    2559 flags.go:64] FLAG: --runonce="false"
Sep 19 08:20:49.729130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720955    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720959    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720962    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720965    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720968    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720972    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720975    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720978    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720981    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720985    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720988    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720991    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720994    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.720998    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721001    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721006    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721009    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721012    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721017    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721020    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721023    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721027    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721032    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721035    2559 flags.go:64] FLAG: --v="2"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721039    2559 flags.go:64] FLAG: --version="false"
Sep 19 08:20:49.729808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721044    2559 flags.go:64] FLAG: --vmodule=""
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721048    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721051    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721193    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721197    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721200    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721203    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721207    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721211    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721216    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721222    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721226    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721229    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721234    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721238    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721241    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721245    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721248    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721251    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721254    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.730556 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721257    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721261    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721265    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721268    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721271    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721274    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721277    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721280    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721283    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721288    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721291    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721293    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721296    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721299    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721302    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721305    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721308    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721311    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721314    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721317    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.731125 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721319    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721322    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721343    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721346    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721349    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721351    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721354    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721357    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721360    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721363    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721366    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721369    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721372    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721375    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721379    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721382    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721385    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721387    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721390    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.731852 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721393    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721396    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721400    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721403    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721406    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721408    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721411    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721414    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721417    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721420    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721423    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721426    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721429    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721432    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721435    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721440    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721451    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721455    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721458    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721461    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.732605 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721464    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721467    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721470    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721473    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721476    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721479    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721482    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721486    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721492    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721497    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721501    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721504    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721508    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721511    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721516    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721519    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721522    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721525    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721528    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.733183 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721531    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721533    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721536    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.721539    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.721545    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.729009    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.729027    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.729048    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.729059    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729123    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729129    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729132    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729136    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729139    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729142    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.733714 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729145    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729148    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729151    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729154    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729157    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729160    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729163    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729165    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729168    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729171    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729176    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729180    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729183    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729186    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729190    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729193    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729196    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729199    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729202    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729206    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.734123 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729209    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729212    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729215    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729220    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729224    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729228    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729231    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729235    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729237    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729240    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729243    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729246    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729249    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729252    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729255    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729258    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729261    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729263    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729266    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.734696 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729269    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729272    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729275    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729278    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729281    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729284    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729286    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729289    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729293    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729296    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729299    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729301    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729304    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729307    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729310    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729313    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729316    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729320    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729323    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729325    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.735311 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729348    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729351    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729354    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729357    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729360    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729363    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729366    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729369    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729372    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729374    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729377    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729380    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729383    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729386    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729389    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729394    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729398    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729401    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729403    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.735932 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729406    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729409    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729412    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729415    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729418    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729421    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729423    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729426    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729429    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729433    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729435    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729439    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729442    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729445    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729448    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.736915 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.729453    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729576    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729581    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729584    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729588    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729591    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729594    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729597    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729599    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729602    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729605    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729608    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729611    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729613    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729616    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729619    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729622    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729625    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729629    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729634    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.737490 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729637    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729639    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729642    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729646    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729648    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729651    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729654    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729657    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729660    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729663    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729666    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729669    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729672    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729674    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729677    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729680    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729682    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729685    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729688    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729690    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.738038 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729693    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729697    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729701    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729705    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729708    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729711    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729715    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729718    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729721    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729724    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729727    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729729    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729733    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729736    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729738    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729741    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729744    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729747    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729750    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.738668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729753    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729755    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729758    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729762    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729764    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729767    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729770    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729772    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729775    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729778    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729781    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729783    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729786    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729789    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729791    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729797    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729800    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729803    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729806    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.739433 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729811    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729814    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729818    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729821    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729823    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729826    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729829    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729832    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729835    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729838    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729840    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729843    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729846    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729849    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729851    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729854    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729857    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729863    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729867    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729869    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.739943 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729872    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.740499 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:49.729875    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.740499 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.729880    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:49.740499 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.729998    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 19 08:20:49.740499 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.734123    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 19 08:20:49.740499 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.734888    2559 server.go:1019] "Starting client certificate rotation"
Sep 19 08:20:49.740499 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.735020    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:49.740499 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.735068    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:49.740499 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.736939    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:49.740499 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.738154    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:49.743395 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.743304    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 19 08:20:49.744738 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.744726    2559 log.go:25] "Validated CRI v1 image API"
Sep 19 08:20:49.746534 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.746519    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 19 08:20:49.748922 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.748896    2559 fs.go:135] Filesystem UUIDs: map[2cca6555-63c7-4013-9622-ba35a00ce8f6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 aa17ab1c-7549-47a6-9849-8bb624081474:/dev/nvme0n1p3]
Sep 19 08:20:49.749042 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.748919    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 19 08:20:49.753821 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.753682    2559 manager.go:216] Machine: {Timestamp:2026-09-19 08:20:49.753079702 +0000 UTC m=+2.163602697 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097600 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d4ff36502211eb7806e6da3bcd2c2 SystemUUID:ec2d4ff3-6502-211e-b780-6e6da3bcd2c2 BootID:b02560c8-99ed-4e98-bf07-3eea1d0bd17e Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:57:89:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:57:89:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:b5:7d:27:15:7b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 19 08:20:49.753821 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.753807    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 19 08:20:49.754018 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.753920    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 19 08:20:49.755950 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.755916    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 19 08:20:49.756140 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.755954    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-26.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 19 08:20:49.757260 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.757247    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 19 08:20:49.757319 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.757269    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 19 08:20:49.757319 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.757282    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:49.757319 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.757299    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:49.757502 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.757430    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 19 08:20:49.761654 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.761633    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:49.761815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.761803    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 19 08:20:49.761877 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.761853    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 19 08:20:49.762624 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.762607    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 19 08:20:49.762704 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.762645    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 19 08:20:49.762704 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.762660    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 19 08:20:49.762704 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.762682    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 19 08:20:49.763575 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.763556    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:49.763665 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.763601    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:49.765833 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.765813    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 19 08:20:49.767943 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.767907    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:49.770401 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.770381    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 19 08:20:49.770473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.770417    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 19 08:20:49.771647 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771619    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 19 08:20:49.771647 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771648    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 19 08:20:49.771762 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771659    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 19 08:20:49.771762 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771673    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 19 08:20:49.771762 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771685    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 19 08:20:49.771762 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771691    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 19 08:20:49.771762 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771709    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 19 08:20:49.771762 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771717    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 19 08:20:49.771762 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771728    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 19 08:20:49.771762 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771738    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 19 08:20:49.771762 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771752    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 19 08:20:49.771762 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771758    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 19 08:20:49.772065 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771787    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 19 08:20:49.772065 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.771793    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 19 08:20:49.774581 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.774560    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 19 08:20:49.774652 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.774585    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-26.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 19 08:20:49.774652 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.774589    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 19 08:20:49.775894 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.775881    2559 server.go:1268] "Started kubelet"
Sep 19 08:20:49.776014 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.775956    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 19 08:20:49.776121 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.776032    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 19 08:20:49.776121 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.775980    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 19 08:20:49.776991 ip-10-0-134-26 systemd[1]: Started Kubernetes Kubelet.
Sep 19 08:20:49.777396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.777046    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 19 08:20:49.777396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.777372    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 19 08:20:49.781184 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.781155    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 19 08:20:49.782613 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.782597    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 19 08:20:49.782680 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.782609    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:49.782732 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.782677    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Sep 19 08:20:49.783205 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783131    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 19 08:20:49.783205 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783135    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 19 08:20:49.783205 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783157    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 19 08:20:49.783422 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783293    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 19 08:20:49.783422 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783303    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 19 08:20:49.783587 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783572    2559 factory.go:55] Registering systemd factory
Sep 19 08:20:49.783637 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783592    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 19 08:20:49.783877 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783862    2559 factory.go:153] Registering CRI-O factory
Sep 19 08:20:49.783947 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783880    2559 factory.go:223] Registration of the crio container factory successfully
Sep 19 08:20:49.783947 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783942    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 19 08:20:49.784044 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783961    2559 factory.go:103] Registering Raw factory
Sep 19 08:20:49.784044 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.783978    2559 manager.go:1196] Started watching for new ooms in manager
Sep 19 08:20:49.784580 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.784371    2559 manager.go:349] Starting recovery of all containers
Sep 19 08:20:49.785339 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.785316    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 19 08:20:49.785446 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.785426    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 19 08:20:49.790769 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.790745    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-btbgg"
Sep 19 08:20:49.813260 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.813239    2559 manager.go:354] Recovery completed
Sep 19 08:20:49.813383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.813353    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-btbgg"
Sep 19 08:20:49.820154 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.820137    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:49.822330 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.822309    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:49.822412 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.822348    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:49.822412 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.822363    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:49.823057 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.823038    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 19 08:20:49.823057 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.823052    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:49.823057 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.823056    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 19 08:20:49.823254 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.823084    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:49.823890 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.823870    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:49.823969 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.823908    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:49.823969 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.823924    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:49.825306 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.825290    2559 policy_none.go:49] "None policy: Start"
Sep 19 08:20:49.825412 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.825310    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 19 08:20:49.825412 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.825326    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 19 08:20:49.827522 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.827506    2559 policy_none.go:47] "Start"
Sep 19 08:20:49.868951 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.868937    2559 manager.go:337] "Starting Device Plugin manager"
Sep 19 08:20:49.869075 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.869038    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 19 08:20:49.869075 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.869054    2559 server.go:87] "Starting device plugin registration server"
Sep 19 08:20:49.869410 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.869385    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 19 08:20:49.869471 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.869442    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 19 08:20:49.869611 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.869586    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 19 08:20:49.869723 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.869690    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 19 08:20:49.869723 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.869698    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 19 08:20:49.870218 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.870196    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 19 08:20:49.870326 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.870247    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-26.ec2.internal\" not found"
Sep 19 08:20:49.930121 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.930082    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 19 08:20:49.931505 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.931487    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 19 08:20:49.931574 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.931564    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 19 08:20:49.931624 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.931597    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 19 08:20:49.931673 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.931649    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 19 08:20:49.933967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.933939    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:49.970407 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.970331    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:49.971380 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.971364    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:49.971462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.971396    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:49.971462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.971412    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:49.971462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.971441    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-26.ec2.internal"
Sep 19 08:20:49.980722 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:49.980704    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-26.ec2.internal"
Sep 19 08:20:49.980805 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.980726    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-26.ec2.internal\": node \"ip-10-0-134-26.ec2.internal\" not found"
Sep 19 08:20:49.999888 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:49.999860    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Sep 19 08:20:50.032722 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.032688    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"]
Sep 19 08:20:50.032808 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.032771    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:50.035287 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.035254    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:50.035355 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.035303    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:50.035355 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.035319    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:50.037858 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.037844    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:50.038007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.037990    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.038051 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.038025    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:50.038776 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.038759    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:50.038856 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.038805    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:50.038856 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.038819    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:50.038856 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.038760    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:50.038974 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.038872    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:50.038974 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.038885    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:50.041221 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.041206    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.041274 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.041233    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:50.042022 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.042003    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:50.042139 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.042030    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:50.042139 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.042043    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:50.055918 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.055898    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-26.ec2.internal\" not found" node="ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.059700 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.059684    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-26.ec2.internal\" not found" node="ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.084895 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.084874    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/768fc03a5f9fae3a346f862bcd558866-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"768fc03a5f9fae3a346f862bcd558866\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.084991 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.084902    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/768fc03a5f9fae3a346f862bcd558866-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"768fc03a5f9fae3a346f862bcd558866\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.084991 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.084933    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a7baa0c5fc4f5272bafad3e803d0c95-config\") pod \"kube-apiserver-proxy-ip-10-0-134-26.ec2.internal\" (UID: \"3a7baa0c5fc4f5272bafad3e803d0c95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.100597 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.100574    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Sep 19 08:20:50.185473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.185440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a7baa0c5fc4f5272bafad3e803d0c95-config\") pod \"kube-apiserver-proxy-ip-10-0-134-26.ec2.internal\" (UID: \"3a7baa0c5fc4f5272bafad3e803d0c95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.185473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.185477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/768fc03a5f9fae3a346f862bcd558866-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"768fc03a5f9fae3a346f862bcd558866\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.185703 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.185497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/768fc03a5f9fae3a346f862bcd558866-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"768fc03a5f9fae3a346f862bcd558866\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.185703 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.185528    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/768fc03a5f9fae3a346f862bcd558866-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"768fc03a5f9fae3a346f862bcd558866\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.185703 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.185534    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3a7baa0c5fc4f5272bafad3e803d0c95-config\") pod \"kube-apiserver-proxy-ip-10-0-134-26.ec2.internal\" (UID: \"3a7baa0c5fc4f5272bafad3e803d0c95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.185703 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.185547    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/768fc03a5f9fae3a346f862bcd558866-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal\" (UID: \"768fc03a5f9fae3a346f862bcd558866\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.201405 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.201379    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Sep 19 08:20:50.302144 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.302046    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Sep 19 08:20:50.362876 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.362853    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.363005 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.362857    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.402379 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.402338    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Sep 19 08:20:50.503069 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.503015    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Sep 19 08:20:50.604128 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.604027    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-26.ec2.internal\" not found"
Sep 19 08:20:50.618633 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.618608    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:50.691163 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.691130    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:50.734447 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.734411    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 19 08:20:50.735183 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.734582    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 08:20:50.735183 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.734582    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 08:20:50.735183 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.734588    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 08:20:50.763920 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.763893    2559 apiserver.go:52] "Watching apiserver"
Sep 19 08:20:50.783361 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.783332    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:50.783361 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.783352    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.783547 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.783378    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 19 08:20:50.785872 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.785340    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-pv57b","openshift-multus/network-metrics-daemon-7qn5v","openshift-ovn-kubernetes/ovnkube-node-hnft5","kube-system/konnectivity-agent-h46xh","openshift-cluster-node-tuning-operator/tuned-tnb9c","openshift-dns/node-resolver-xn4mf","openshift-network-diagnostics/network-check-target-gcm28","openshift-network-operator/iptables-alerter-wnn76","kube-system/global-pull-secret-syncer-xvkt5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns","openshift-image-registry/node-ca-lcmrp","openshift-multus/multus-additional-cni-plugins-g2x69"]
Sep 19 08:20:50.787931 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.787911    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.790011 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.789988    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:50.790130 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.790078    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:20:50.790394 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.790376    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 19 08:20:50.790501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.790485    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 19 08:20:50.790582 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.790563    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 19 08:20:50.791056 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.791038    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6t8md\""
Sep 19 08:20:50.791137 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.791037    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 19 08:20:50.792210 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.792190    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.794232 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.794211    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:20:50.794622 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.794607    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 19 08:20:50.795789 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.795769    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 19 08:20:50.795886 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.795818    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 19 08:20:50.795886 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.795838    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 19 08:20:50.795886 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.795880    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 19 08:20:50.796021 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.795881    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6qqrt\""
Sep 19 08:20:50.796021 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.795863    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 19 08:20:50.796637 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.796591    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.796970 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.796954    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-86gfp\""
Sep 19 08:20:50.797035 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.797010    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 19 08:20:50.797081 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.797053    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:50.797081 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.797075    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 19 08:20:50.797176 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.797132    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"
Sep 19 08:20:50.798774 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.798758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xn4mf"
Sep 19 08:20:50.799282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.799248    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zpwjw\""
Sep 19 08:20:50.799388 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.799283    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:50.799388 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.799293    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:50.799677 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.799661    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:50.800980 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.800959    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:20:50.801069 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.801037    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gtngz\""
Sep 19 08:20:50.801129 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.801060    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:20:50.801297 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.801280    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 19 08:20:50.801353 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.801329    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 19 08:20:50.803277 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.803246    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wnn76"
Sep 19 08:20:50.805510 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.805492    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:50.805609 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.805529    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:50.805669 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.805646    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 19 08:20:50.805789 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.805771    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lnfmt\""
Sep 19 08:20:50.805892 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.805879    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:50.805953 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.805937    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:20:50.806455 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.806438    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:50.811024 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.811009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.812424 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.812406    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lcmrp"
Sep 19 08:20:50.813380 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.813360    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 19 08:20:50.813380 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.813377    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-74b6j\""
Sep 19 08:20:50.813523 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.813410    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 19 08:20:50.813523 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.813475    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 19 08:20:50.814410 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.814392    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal"]
Sep 19 08:20:50.814493 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.814425    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal"]
Sep 19 08:20:50.814533 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.814497    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.814614 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.814602    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 19 08:20:50.814950 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.814936    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 19 08:20:50.814997 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.814960    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h8qhq\""
Sep 19 08:20:50.814997 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.814980    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 19 08:20:50.816595 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.816580    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 19 08:20:50.817186 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.817162    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mwxz5\""
Sep 19 08:20:50.817584 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.817561    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qmqsg"
Sep 19 08:20:50.817953 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.817780    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 19 08:20:50.818914 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.818839    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 08:15:49 +0000 UTC" deadline="2028-05-22 03:06:01.329778329 +0000 UTC"
Sep 19 08:20:50.819052 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.819038    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14658h45m10.510746553s"
Sep 19 08:20:50.826285 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.826266    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qmqsg"
Sep 19 08:20:50.883895 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.883874    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 19 08:20:50.890583 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890560    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-cni-bin\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.890701 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890588    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/078bb266-8297-48c9-a45b-2d47fb7cbd45-kubelet-config\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:50.890701 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-log-socket\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.890701 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.890701 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-socket-dir-parent\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.890701 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-run-k8s-cni-cncf-io\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.890851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890695    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s98sk\" (UniqueName: \"kubernetes.io/projected/007b6a79-68a9-4bf6-8d49-897b5151c149-kube-api-access-s98sk\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.890851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-systemd-units\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.890851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890733    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-var-lib-kubelet\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.890851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d618e83-8b27-4401-94d2-1feaa04c634f-host\") pod \"node-ca-lcmrp\" (UID: \"2d618e83-8b27-4401-94d2-1feaa04c634f\") " pod="openshift-image-registry/node-ca-lcmrp"
Sep 19 08:20:50.890851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890767    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-hostroot\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.890851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-run-multus-certs\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.890851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af211ba6-f358-4565-a7dd-37eb2337ea07-tmp-dir\") pod \"node-resolver-xn4mf\" (UID: \"af211ba6-f358-4565-a7dd-37eb2337ea07\") " pod="openshift-dns/node-resolver-xn4mf"
Sep 19 08:20:50.890851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/338a6f68-5bed-457b-b1f2-23f4e68922d6-ovnkube-config\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.890851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890825    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-host\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.890851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890842    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:50.891119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-conf-dir\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.891119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890883    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-sys-fs\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.891119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890921    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-daemon-config\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.891119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890968    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-slash\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.891119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.890994    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/338a6f68-5bed-457b-b1f2-23f4e68922d6-ovnkube-script-lib\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.891119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891023    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-modprobe-d\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.891119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891044    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-etc-kubernetes\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.891119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891060    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fb5bd3b-10c8-41fb-b8b1-d14846935295-system-cni-dir\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.891119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-system-cni-dir\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.891119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891089    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-run-netns\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.891119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891116    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891138    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/338a6f68-5bed-457b-b1f2-23f4e68922d6-ovn-node-metrics-cert\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891152    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvxth\" (UniqueName: \"kubernetes.io/projected/338a6f68-5bed-457b-b1f2-23f4e68922d6-kube-api-access-jvxth\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fb5bd3b-10c8-41fb-b8b1-d14846935295-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/955095d1-46e8-49b9-a667-a40d58fa50ef-agent-certs\") pod \"konnectivity-agent-h46xh\" (UID: \"955095d1-46e8-49b9-a667-a40d58fa50ef\") " pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891226    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-etc-openvswitch\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-cnibin\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891278    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fb5bd3b-10c8-41fb-b8b1-d14846935295-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-device-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891310    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d618e83-8b27-4401-94d2-1feaa04c634f-serviceca\") pod \"node-ca-lcmrp\" (UID: \"2d618e83-8b27-4401-94d2-1feaa04c634f\") " pod="openshift-image-registry/node-ca-lcmrp"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891323    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-run-systemd\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-var-lib-cni-bin\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pdb9\" (UniqueName: \"kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9\") pod \"network-check-target-gcm28\" (UID: \"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1\") " pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9tv\" (UniqueName: \"kubernetes.io/projected/2d618e83-8b27-4401-94d2-1feaa04c634f-kube-api-access-sc9tv\") pod \"node-ca-lcmrp\" (UID: \"2d618e83-8b27-4401-94d2-1feaa04c634f\") " pod="openshift-image-registry/node-ca-lcmrp"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891405    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-tuned\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.891480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4rw\" (UniqueName: \"kubernetes.io/projected/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-kube-api-access-4l4rw\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891481    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ac0106e-5a03-4fcb-a733-5fd551497dc0-iptables-alerter-script\") pod \"iptables-alerter-wnn76\" (UID: \"8ac0106e-5a03-4fcb-a733-5fd551497dc0\") " pod="openshift-network-operator/iptables-alerter-wnn76"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891538    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-run-openvswitch\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891563    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/007b6a79-68a9-4bf6-8d49-897b5151c149-cni-binary-copy\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891587    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fb5bd3b-10c8-41fb-b8b1-d14846935295-cnibin\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fb5bd3b-10c8-41fb-b8b1-d14846935295-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891634    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fb5bd3b-10c8-41fb-b8b1-d14846935295-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891657    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-kubernetes\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891680    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-sys\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891720    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-lib-modules\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891734    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af211ba6-f358-4565-a7dd-37eb2337ea07-hosts-file\") pod \"node-resolver-xn4mf\" (UID: \"af211ba6-f358-4565-a7dd-37eb2337ea07\") " pod="openshift-dns/node-resolver-xn4mf"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891797    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891818    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-kubelet\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891834    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-run-ovn\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891848    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-sysconfig\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.891996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/078bb266-8297-48c9-a45b-2d47fb7cbd45-dbus\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891886    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvngz\" (UniqueName: \"kubernetes.io/projected/9178e60e-1f54-461c-8a40-1b8c3300b64c-kube-api-access-mvngz\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891902    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-sysctl-conf\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891928    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnndz\" (UniqueName: \"kubernetes.io/projected/8ac0106e-5a03-4fcb-a733-5fd551497dc0-kube-api-access-tnndz\") pod \"iptables-alerter-wnn76\" (UID: \"8ac0106e-5a03-4fcb-a733-5fd551497dc0\") " pod="openshift-network-operator/iptables-alerter-wnn76"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891951    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxdjx\" (UniqueName: \"kubernetes.io/projected/9fb5bd3b-10c8-41fb-b8b1-d14846935295-kube-api-access-gxdjx\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891968    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2ds7\" (UniqueName: \"kubernetes.io/projected/af211ba6-f358-4565-a7dd-37eb2337ea07-kube-api-access-b2ds7\") pod \"node-resolver-xn4mf\" (UID: \"af211ba6-f358-4565-a7dd-37eb2337ea07\") " pod="openshift-dns/node-resolver-xn4mf"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.891989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-registration-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/338a6f68-5bed-457b-b1f2-23f4e68922d6-env-overrides\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-systemd\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892032    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-var-lib-cni-multus\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892078    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-sysctl-d\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-run\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ac0106e-5a03-4fcb-a733-5fd551497dc0-host-slash\") pod \"iptables-alerter-wnn76\" (UID: \"8ac0106e-5a03-4fcb-a733-5fd551497dc0\") " pod="openshift-network-operator/iptables-alerter-wnn76"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-run-netns\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892185    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-var-lib-kubelet\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.892501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892210    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/955095d1-46e8-49b9-a667-a40d58fa50ef-konnectivity-ca\") pod \"konnectivity-agent-h46xh\" (UID: \"955095d1-46e8-49b9-a667-a40d58fa50ef\") " pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:20:50.893110 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-var-lib-openvswitch\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.893110 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892282    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-node-log\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.893110 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892316    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-cni-netd\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.893110 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892330    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-tmp\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.893110 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-cni-dir\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.893110 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-socket-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.893110 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.892419    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7jl7\" (UniqueName: \"kubernetes.io/projected/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-kube-api-access-j7jl7\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.993199 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993174    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxdjx\" (UniqueName: \"kubernetes.io/projected/9fb5bd3b-10c8-41fb-b8b1-d14846935295-kube-api-access-gxdjx\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.993371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993204    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2ds7\" (UniqueName: \"kubernetes.io/projected/af211ba6-f358-4565-a7dd-37eb2337ea07-kube-api-access-b2ds7\") pod \"node-resolver-xn4mf\" (UID: \"af211ba6-f358-4565-a7dd-37eb2337ea07\") " pod="openshift-dns/node-resolver-xn4mf"
Sep 19 08:20:50.993371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993221    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-registration-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.993371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993236    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/338a6f68-5bed-457b-b1f2-23f4e68922d6-env-overrides\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.993371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-systemd\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.993371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993322    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-systemd\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.993371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993337    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-registration-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.993582 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993491    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-var-lib-cni-multus\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.993582 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993520    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.993582 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-sysctl-d\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.993687 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-var-lib-cni-multus\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.993687 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993607    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.993687 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993622    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-run\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.993687 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-sysctl-d\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.993687 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993651    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ac0106e-5a03-4fcb-a733-5fd551497dc0-host-slash\") pod \"iptables-alerter-wnn76\" (UID: \"8ac0106e-5a03-4fcb-a733-5fd551497dc0\") " pod="openshift-network-operator/iptables-alerter-wnn76"
Sep 19 08:20:50.993687 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993659    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-run\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.993687 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-run-netns\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ac0106e-5a03-4fcb-a733-5fd551497dc0-host-slash\") pod \"iptables-alerter-wnn76\" (UID: \"8ac0106e-5a03-4fcb-a733-5fd551497dc0\") " pod="openshift-network-operator/iptables-alerter-wnn76"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993701    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-var-lib-kubelet\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993701    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/338a6f68-5bed-457b-b1f2-23f4e68922d6-env-overrides\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993725    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/955095d1-46e8-49b9-a667-a40d58fa50ef-konnectivity-ca\") pod \"konnectivity-agent-h46xh\" (UID: \"955095d1-46e8-49b9-a667-a40d58fa50ef\") " pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-run-netns\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993749    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-var-lib-kubelet\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-var-lib-openvswitch\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-node-log\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993789    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-cni-netd\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-tmp\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993817    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-cni-dir\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993839    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-cni-netd\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-node-log\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-socket-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-var-lib-openvswitch\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993879    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7jl7\" (UniqueName: \"kubernetes.io/projected/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-kube-api-access-j7jl7\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993890    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-cni-dir\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.993909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-cni-bin\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/078bb266-8297-48c9-a45b-2d47fb7cbd45-kubelet-config\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-socket-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993977    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/078bb266-8297-48c9-a45b-2d47fb7cbd45-kubelet-config\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.993991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-cni-bin\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994038    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-log-socket\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-log-socket\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-socket-dir-parent\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994128    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-run-k8s-cni-cncf-io\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994135    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994136    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s98sk\" (UniqueName: \"kubernetes.io/projected/007b6a79-68a9-4bf6-8d49-897b5151c149-kube-api-access-s98sk\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994165    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-systemd-units\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994169    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-run-k8s-cni-cncf-io\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-var-lib-kubelet\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994197    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d618e83-8b27-4401-94d2-1feaa04c634f-host\") pod \"node-ca-lcmrp\" (UID: \"2d618e83-8b27-4401-94d2-1feaa04c634f\") " pod="openshift-image-registry/node-ca-lcmrp"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-socket-dir-parent\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.994740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994219    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-hostroot\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994235    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-var-lib-kubelet\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994248    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-systemd-units\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/955095d1-46e8-49b9-a667-a40d58fa50ef-konnectivity-ca\") pod \"konnectivity-agent-h46xh\" (UID: \"955095d1-46e8-49b9-a667-a40d58fa50ef\") " pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994276    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-run-multus-certs\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-hostroot\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-run-multus-certs\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994278    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d618e83-8b27-4401-94d2-1feaa04c634f-host\") pod \"node-ca-lcmrp\" (UID: \"2d618e83-8b27-4401-94d2-1feaa04c634f\") " pod="openshift-image-registry/node-ca-lcmrp"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af211ba6-f358-4565-a7dd-37eb2337ea07-tmp-dir\") pod \"node-resolver-xn4mf\" (UID: \"af211ba6-f358-4565-a7dd-37eb2337ea07\") " pod="openshift-dns/node-resolver-xn4mf"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/338a6f68-5bed-457b-b1f2-23f4e68922d6-ovnkube-config\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-host\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994384    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994406    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-host\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-conf-dir\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994449    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-conf-dir\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-sys-fs\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.994497    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-daemon-config\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.995556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994524    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-slash\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/338a6f68-5bed-457b-b1f2-23f4e68922d6-ovnkube-script-lib\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994567    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-sys-fs\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.994588    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret podName:078bb266-8297-48c9-a45b-2d47fb7cbd45 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:51.494550566 +0000 UTC m=+3.905073573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret") pod "global-pull-secret-syncer-xvkt5" (UID: "078bb266-8297-48c9-a45b-2d47fb7cbd45") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-modprobe-d\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af211ba6-f358-4565-a7dd-37eb2337ea07-tmp-dir\") pod \"node-resolver-xn4mf\" (UID: \"af211ba6-f358-4565-a7dd-37eb2337ea07\") " pod="openshift-dns/node-resolver-xn4mf"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-modprobe-d\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-slash\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-etc-kubernetes\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994814    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fb5bd3b-10c8-41fb-b8b1-d14846935295-system-cni-dir\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/338a6f68-5bed-457b-b1f2-23f4e68922d6-ovnkube-config\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994837    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-system-cni-dir\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-run-netns\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994869    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-etc-kubernetes\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994878    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9fb5bd3b-10c8-41fb-b8b1-d14846935295-system-cni-dir\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994886    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/338a6f68-5bed-457b-b1f2-23f4e68922d6-ovn-node-metrics-cert\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.996383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994925    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-system-cni-dir\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994929    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-run-netns\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994940    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxth\" (UniqueName: \"kubernetes.io/projected/338a6f68-5bed-457b-b1f2-23f4e68922d6-kube-api-access-jvxth\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fb5bd3b-10c8-41fb-b8b1-d14846935295-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994970    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.994994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/955095d1-46e8-49b9-a667-a40d58fa50ef-agent-certs\") pod \"konnectivity-agent-h46xh\" (UID: \"955095d1-46e8-49b9-a667-a40d58fa50ef\") " pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995180    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-etc-openvswitch\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995188    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/007b6a79-68a9-4bf6-8d49-897b5151c149-multus-daemon-config\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995211    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-cnibin\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995272    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fb5bd3b-10c8-41fb-b8b1-d14846935295-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995289    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-cnibin\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995341    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-etc-openvswitch\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-device-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995402    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d618e83-8b27-4401-94d2-1feaa04c634f-serviceca\") pod \"node-ca-lcmrp\" (UID: \"2d618e83-8b27-4401-94d2-1feaa04c634f\") " pod="openshift-image-registry/node-ca-lcmrp"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-run-systemd\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-var-lib-cni-bin\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdb9\" (UniqueName: \"kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9\") pod \"network-check-target-gcm28\" (UID: \"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1\") " pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:20:50.997192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995485    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-device-dir\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995502    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9tv\" (UniqueName: \"kubernetes.io/projected/2d618e83-8b27-4401-94d2-1feaa04c634f-kube-api-access-sc9tv\") pod \"node-ca-lcmrp\" (UID: \"2d618e83-8b27-4401-94d2-1feaa04c634f\") " pod="openshift-image-registry/node-ca-lcmrp"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995558    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/007b6a79-68a9-4bf6-8d49-897b5151c149-host-var-lib-cni-bin\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-tuned\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4rw\" (UniqueName: \"kubernetes.io/projected/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-kube-api-access-4l4rw\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ac0106e-5a03-4fcb-a733-5fd551497dc0-iptables-alerter-script\") pod \"iptables-alerter-wnn76\" (UID: \"8ac0106e-5a03-4fcb-a733-5fd551497dc0\") " pod="openshift-network-operator/iptables-alerter-wnn76"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-run-openvswitch\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/007b6a79-68a9-4bf6-8d49-897b5151c149-cni-binary-copy\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fb5bd3b-10c8-41fb-b8b1-d14846935295-cnibin\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995740    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fb5bd3b-10c8-41fb-b8b1-d14846935295-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995765    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fb5bd3b-10c8-41fb-b8b1-d14846935295-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995790    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-kubernetes\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995813    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-sys\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995836    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-lib-modules\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995848    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9fb5bd3b-10c8-41fb-b8b1-d14846935295-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995859    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995911    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9fb5bd3b-10c8-41fb-b8b1-d14846935295-cnibin\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af211ba6-f358-4565-a7dd-37eb2337ea07-hosts-file\") pod \"node-resolver-xn4mf\" (UID: \"af211ba6-f358-4565-a7dd-37eb2337ea07\") " pod="openshift-dns/node-resolver-xn4mf"
Sep 19 08:20:50.997891 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.995934    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d618e83-8b27-4401-94d2-1feaa04c634f-serviceca\") pod \"node-ca-lcmrp\" (UID: \"2d618e83-8b27-4401-94d2-1feaa04c634f\") " pod="openshift-image-registry/node-ca-lcmrp"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.995949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:50.995992    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs podName:9178e60e-1f54-461c-8a40-1b8c3300b64c nodeName:}" failed. No retries permitted until 2026-09-19 08:20:51.495975146 +0000 UTC m=+3.906498134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs") pod "network-metrics-daemon-7qn5v" (UID: "9178e60e-1f54-461c-8a40-1b8c3300b64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996010    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-kubelet\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996014    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996036    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-run-ovn\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-sysconfig\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/078bb266-8297-48c9-a45b-2d47fb7cbd45-dbus\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996124    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-run-openvswitch\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996131    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvngz\" (UniqueName: \"kubernetes.io/projected/9178e60e-1f54-461c-8a40-1b8c3300b64c-kube-api-access-mvngz\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-sysctl-conf\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnndz\" (UniqueName: \"kubernetes.io/projected/8ac0106e-5a03-4fcb-a733-5fd551497dc0-kube-api-access-tnndz\") pod \"iptables-alerter-wnn76\" (UID: \"8ac0106e-5a03-4fcb-a733-5fd551497dc0\") " pod="openshift-network-operator/iptables-alerter-wnn76"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-host-kubelet\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996391    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/007b6a79-68a9-4bf6-8d49-897b5151c149-cni-binary-copy\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996402    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-run-ovn\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996468    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-sysconfig\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.998643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996058    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/af211ba6-f358-4565-a7dd-37eb2337ea07-hosts-file\") pod \"node-resolver-xn4mf\" (UID: \"af211ba6-f358-4565-a7dd-37eb2337ea07\") " pod="openshift-dns/node-resolver-xn4mf"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996577    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/078bb266-8297-48c9-a45b-2d47fb7cbd45-dbus\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996603    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ac0106e-5a03-4fcb-a733-5fd551497dc0-iptables-alerter-script\") pod \"iptables-alerter-wnn76\" (UID: \"8ac0106e-5a03-4fcb-a733-5fd551497dc0\") " pod="openshift-network-operator/iptables-alerter-wnn76"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996619    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-sysctl-conf\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996681    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-kubernetes\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9fb5bd3b-10c8-41fb-b8b1-d14846935295-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996876    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-sys\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/338a6f68-5bed-457b-b1f2-23f4e68922d6-run-systemd\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.996958    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-lib-modules\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.997077    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/338a6f68-5bed-457b-b1f2-23f4e68922d6-ovnkube-script-lib\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.997109    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9fb5bd3b-10c8-41fb-b8b1-d14846935295-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.997178    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9fb5bd3b-10c8-41fb-b8b1-d14846935295-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.997696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-tmp\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.998899    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/955095d1-46e8-49b9-a667-a40d58fa50ef-agent-certs\") pod \"konnectivity-agent-h46xh\" (UID: \"955095d1-46e8-49b9-a667-a40d58fa50ef\") " pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.999059    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/338a6f68-5bed-457b-b1f2-23f4e68922d6-ovn-node-metrics-cert\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:50.999177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:50.999108    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-etc-tuned\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:51.000680 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:51.000655    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768fc03a5f9fae3a346f862bcd558866.slice/crio-26a7ba7a434d1695e0bd9836c4b19a9ec51798feb605f5c4d9214403cf0267d2 WatchSource:0}: Error finding container 26a7ba7a434d1695e0bd9836c4b19a9ec51798feb605f5c4d9214403cf0267d2: Status 404 returned error can't find the container with id 26a7ba7a434d1695e0bd9836c4b19a9ec51798feb605f5c4d9214403cf0267d2
Sep 19 08:20:51.000892 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:51.000876    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7baa0c5fc4f5272bafad3e803d0c95.slice/crio-7c2a8d16d6256d79413c4a466b4b14d4b8e5e30a6796e91cd5daeecd269170d4 WatchSource:0}: Error finding container 7c2a8d16d6256d79413c4a466b4b14d4b8e5e30a6796e91cd5daeecd269170d4: Status 404 returned error can't find the container with id 7c2a8d16d6256d79413c4a466b4b14d4b8e5e30a6796e91cd5daeecd269170d4
Sep 19 08:20:51.005286 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.005268    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:20:51.007776 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.007451    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:51.007776 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.007473    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:51.007776 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.007488    2559 projected.go:196] Error preparing data for projected volume kube-api-access-2pdb9 for pod openshift-network-diagnostics/network-check-target-gcm28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:51.007776 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.007548    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9 podName:4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:51.507528981 +0000 UTC m=+3.918051975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2pdb9" (UniqueName: "kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9") pod "network-check-target-gcm28" (UID: "4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:51.007776 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.007699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2ds7\" (UniqueName: \"kubernetes.io/projected/af211ba6-f358-4565-a7dd-37eb2337ea07-kube-api-access-b2ds7\") pod \"node-resolver-xn4mf\" (UID: \"af211ba6-f358-4565-a7dd-37eb2337ea07\") " pod="openshift-dns/node-resolver-xn4mf"
Sep 19 08:20:51.008651 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.008632    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s98sk\" (UniqueName: \"kubernetes.io/projected/007b6a79-68a9-4bf6-8d49-897b5151c149-kube-api-access-s98sk\") pod \"multus-pv57b\" (UID: \"007b6a79-68a9-4bf6-8d49-897b5151c149\") " pod="openshift-multus/multus-pv57b"
Sep 19 08:20:51.008690 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.008668    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxdjx\" (UniqueName: \"kubernetes.io/projected/9fb5bd3b-10c8-41fb-b8b1-d14846935295-kube-api-access-gxdjx\") pod \"multus-additional-cni-plugins-g2x69\" (UID: \"9fb5bd3b-10c8-41fb-b8b1-d14846935295\") " pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:51.009646 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.009621    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7jl7\" (UniqueName: \"kubernetes.io/projected/d3ed0f54-9d5e-4fd0-a672-761615ba1eba-kube-api-access-j7jl7\") pod \"aws-ebs-csi-driver-node-vdlns\" (UID: \"d3ed0f54-9d5e-4fd0-a672-761615ba1eba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:51.010279 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.010238    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvngz\" (UniqueName: \"kubernetes.io/projected/9178e60e-1f54-461c-8a40-1b8c3300b64c-kube-api-access-mvngz\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:51.011005 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.010984    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4rw\" (UniqueName: \"kubernetes.io/projected/bca89dd8-76bb-4909-be5e-91f7f7fa40bc-kube-api-access-4l4rw\") pod \"tuned-tnb9c\" (UID: \"bca89dd8-76bb-4909-be5e-91f7f7fa40bc\") " pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:51.011192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.011176    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9tv\" (UniqueName: \"kubernetes.io/projected/2d618e83-8b27-4401-94d2-1feaa04c634f-kube-api-access-sc9tv\") pod \"node-ca-lcmrp\" (UID: \"2d618e83-8b27-4401-94d2-1feaa04c634f\") " pod="openshift-image-registry/node-ca-lcmrp"
Sep 19 08:20:51.011393 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.011372    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnndz\" (UniqueName: \"kubernetes.io/projected/8ac0106e-5a03-4fcb-a733-5fd551497dc0-kube-api-access-tnndz\") pod \"iptables-alerter-wnn76\" (UID: \"8ac0106e-5a03-4fcb-a733-5fd551497dc0\") " pod="openshift-network-operator/iptables-alerter-wnn76"
Sep 19 08:20:51.011475 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.011462    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxth\" (UniqueName: \"kubernetes.io/projected/338a6f68-5bed-457b-b1f2-23f4e68922d6-kube-api-access-jvxth\") pod \"ovnkube-node-hnft5\" (UID: \"338a6f68-5bed-457b-b1f2-23f4e68922d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:51.020942 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.020924    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:51.128310 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.128224    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pv57b"
Sep 19 08:20:51.139316 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:51.139288    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007b6a79_68a9_4bf6_8d49_897b5151c149.slice/crio-6f25c3bb43e3464f6d73798b95c458b1d0804c30283ba7cba9bb974de1fb5fd7 WatchSource:0}: Error finding container 6f25c3bb43e3464f6d73798b95c458b1d0804c30283ba7cba9bb974de1fb5fd7: Status 404 returned error can't find the container with id 6f25c3bb43e3464f6d73798b95c458b1d0804c30283ba7cba9bb974de1fb5fd7
Sep 19 08:20:51.142603 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.142585    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:20:51.150463 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:51.150437    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338a6f68_5bed_457b_b1f2_23f4e68922d6.slice/crio-f547ca62c9ef310756e0ba1bff141fecec75ce2ea78a38ce406711c6702b9ff0 WatchSource:0}: Error finding container f547ca62c9ef310756e0ba1bff141fecec75ce2ea78a38ce406711c6702b9ff0: Status 404 returned error can't find the container with id f547ca62c9ef310756e0ba1bff141fecec75ce2ea78a38ce406711c6702b9ff0
Sep 19 08:20:51.151148 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.151135    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:20:51.158207 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.158189    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tnb9c"
Sep 19 08:20:51.159668 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:51.159648    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955095d1_46e8_49b9_a667_a40d58fa50ef.slice/crio-1fa8ae896cd98e116141d60b2b0f94d4c0d7f1ad4a9371b2f64e05dca3776b4a WatchSource:0}: Error finding container 1fa8ae896cd98e116141d60b2b0f94d4c0d7f1ad4a9371b2f64e05dca3776b4a: Status 404 returned error can't find the container with id 1fa8ae896cd98e116141d60b2b0f94d4c0d7f1ad4a9371b2f64e05dca3776b4a
Sep 19 08:20:51.167814 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:51.167793    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca89dd8_76bb_4909_be5e_91f7f7fa40bc.slice/crio-0d47e1e550252af7d8af33ec8401f9ddbe3d39502240edd65dbc11e59a61d9cf WatchSource:0}: Error finding container 0d47e1e550252af7d8af33ec8401f9ddbe3d39502240edd65dbc11e59a61d9cf: Status 404 returned error can't find the container with id 0d47e1e550252af7d8af33ec8401f9ddbe3d39502240edd65dbc11e59a61d9cf
Sep 19 08:20:51.179189 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.179172    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xn4mf"
Sep 19 08:20:51.187071 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:51.187052    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf211ba6_f358_4565_a7dd_37eb2337ea07.slice/crio-576b69e8f0f6d3f4df36d3870c09f855243e554d68fa935d4ac480e1c6934f34 WatchSource:0}: Error finding container 576b69e8f0f6d3f4df36d3870c09f855243e554d68fa935d4ac480e1c6934f34: Status 404 returned error can't find the container with id 576b69e8f0f6d3f4df36d3870c09f855243e554d68fa935d4ac480e1c6934f34
Sep 19 08:20:51.195536 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.195519    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wnn76"
Sep 19 08:20:51.203710 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:51.203688    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac0106e_5a03_4fcb_a733_5fd551497dc0.slice/crio-4aff3beb469a5e4a33b870c1c355f8f6ed98e08561fcf9b581d0c5ec523688a6 WatchSource:0}: Error finding container 4aff3beb469a5e4a33b870c1c355f8f6ed98e08561fcf9b581d0c5ec523688a6: Status 404 returned error can't find the container with id 4aff3beb469a5e4a33b870c1c355f8f6ed98e08561fcf9b581d0c5ec523688a6
Sep 19 08:20:51.211236 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.211219    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns"
Sep 19 08:20:51.218967 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:51.218948    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ed0f54_9d5e_4fd0_a672_761615ba1eba.slice/crio-a1da516b1fab7bece964243682ae2a661e11a4ce9c873af16a74ff33aa26d026 WatchSource:0}: Error finding container a1da516b1fab7bece964243682ae2a661e11a4ce9c873af16a74ff33aa26d026: Status 404 returned error can't find the container with id a1da516b1fab7bece964243682ae2a661e11a4ce9c873af16a74ff33aa26d026
Sep 19 08:20:51.225858 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.225844    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lcmrp"
Sep 19 08:20:51.230400 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.230385    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2x69"
Sep 19 08:20:51.234532 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:51.234512    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d618e83_8b27_4401_94d2_1feaa04c634f.slice/crio-fcf6aea05215d3d0149ab2c4cf230a5276b02b4ac50b4fdf34f6fc2102ea9ca9 WatchSource:0}: Error finding container fcf6aea05215d3d0149ab2c4cf230a5276b02b4ac50b4fdf34f6fc2102ea9ca9: Status 404 returned error can't find the container with id fcf6aea05215d3d0149ab2c4cf230a5276b02b4ac50b4fdf34f6fc2102ea9ca9
Sep 19 08:20:51.242074 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:20:51.242055    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb5bd3b_10c8_41fb_b8b1_d14846935295.slice/crio-41b259ef00a934dc861eab6046b06bef78e748f5c59e22b0dc9adad72a2b0426 WatchSource:0}: Error finding container 41b259ef00a934dc861eab6046b06bef78e748f5c59e22b0dc9adad72a2b0426: Status 404 returned error can't find the container with id 41b259ef00a934dc861eab6046b06bef78e748f5c59e22b0dc9adad72a2b0426
Sep 19 08:20:51.500150 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.500068    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:51.500150 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.500144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:51.500413 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.500305    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:51.500413 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.500371    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret podName:078bb266-8297-48c9-a45b-2d47fb7cbd45 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:52.500352278 +0000 UTC m=+4.910875261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret") pod "global-pull-secret-syncer-xvkt5" (UID: "078bb266-8297-48c9-a45b-2d47fb7cbd45") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:51.500537 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.500438    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:51.500537 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.500472    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs podName:9178e60e-1f54-461c-8a40-1b8c3300b64c nodeName:}" failed. No retries permitted until 2026-09-19 08:20:52.500458913 +0000 UTC m=+4.910981892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs") pod "network-metrics-daemon-7qn5v" (UID: "9178e60e-1f54-461c-8a40-1b8c3300b64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:51.600586 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.600542    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdb9\" (UniqueName: \"kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9\") pod \"network-check-target-gcm28\" (UID: \"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1\") " pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:20:51.600767 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.600736    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:51.600767 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.600759    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:51.600873 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.600772    2559 projected.go:196] Error preparing data for projected volume kube-api-access-2pdb9 for pod openshift-network-diagnostics/network-check-target-gcm28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:51.600873 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:51.600833    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9 podName:4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:52.600813763 +0000 UTC m=+5.011336746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pdb9" (UniqueName: "kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9") pod "network-check-target-gcm28" (UID: "4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:51.830171 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.830056    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:50 +0000 UTC" deadline="2028-06-04 12:21:26.544534846 +0000 UTC"
Sep 19 08:20:51.830171 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.830123    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14980h0m34.714415816s"
Sep 19 08:20:51.966103 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.965889    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:51.987546 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:51.987470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tnb9c" event={"ID":"bca89dd8-76bb-4909-be5e-91f7f7fa40bc","Type":"ContainerStarted","Data":"0d47e1e550252af7d8af33ec8401f9ddbe3d39502240edd65dbc11e59a61d9cf"}
Sep 19 08:20:52.025897 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.025821    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h46xh" event={"ID":"955095d1-46e8-49b9-a667-a40d58fa50ef","Type":"ContainerStarted","Data":"1fa8ae896cd98e116141d60b2b0f94d4c0d7f1ad4a9371b2f64e05dca3776b4a"}
Sep 19 08:20:52.051677 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.051610    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pv57b" event={"ID":"007b6a79-68a9-4bf6-8d49-897b5151c149","Type":"ContainerStarted","Data":"6f25c3bb43e3464f6d73798b95c458b1d0804c30283ba7cba9bb974de1fb5fd7"}
Sep 19 08:20:52.076585 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.076443    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns" event={"ID":"d3ed0f54-9d5e-4fd0-a672-761615ba1eba","Type":"ContainerStarted","Data":"a1da516b1fab7bece964243682ae2a661e11a4ce9c873af16a74ff33aa26d026"}
Sep 19 08:20:52.084580 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.084502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wnn76" event={"ID":"8ac0106e-5a03-4fcb-a733-5fd551497dc0","Type":"ContainerStarted","Data":"4aff3beb469a5e4a33b870c1c355f8f6ed98e08561fcf9b581d0c5ec523688a6"}
Sep 19 08:20:52.102031 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.101940    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xn4mf" event={"ID":"af211ba6-f358-4565-a7dd-37eb2337ea07","Type":"ContainerStarted","Data":"576b69e8f0f6d3f4df36d3870c09f855243e554d68fa935d4ac480e1c6934f34"}
Sep 19 08:20:52.111717 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.111685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5" event={"ID":"338a6f68-5bed-457b-b1f2-23f4e68922d6","Type":"ContainerStarted","Data":"f547ca62c9ef310756e0ba1bff141fecec75ce2ea78a38ce406711c6702b9ff0"}
Sep 19 08:20:52.129238 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.129130    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal" event={"ID":"3a7baa0c5fc4f5272bafad3e803d0c95","Type":"ContainerStarted","Data":"7c2a8d16d6256d79413c4a466b4b14d4b8e5e30a6796e91cd5daeecd269170d4"}
Sep 19 08:20:52.141984 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.141949    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal" event={"ID":"768fc03a5f9fae3a346f862bcd558866","Type":"ContainerStarted","Data":"26a7ba7a434d1695e0bd9836c4b19a9ec51798feb605f5c4d9214403cf0267d2"}
Sep 19 08:20:52.153546 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.153511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2x69" event={"ID":"9fb5bd3b-10c8-41fb-b8b1-d14846935295","Type":"ContainerStarted","Data":"41b259ef00a934dc861eab6046b06bef78e748f5c59e22b0dc9adad72a2b0426"}
Sep 19 08:20:52.158023 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.157992    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lcmrp" event={"ID":"2d618e83-8b27-4401-94d2-1feaa04c634f","Type":"ContainerStarted","Data":"fcf6aea05215d3d0149ab2c4cf230a5276b02b4ac50b4fdf34f6fc2102ea9ca9"}
Sep 19 08:20:52.510965 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.510873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:52.510965 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.510954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:52.511185 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:52.511083    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:52.511185 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:52.511145    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs podName:9178e60e-1f54-461c-8a40-1b8c3300b64c nodeName:}" failed. No retries permitted until 2026-09-19 08:20:54.51112576 +0000 UTC m=+6.921648744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs") pod "network-metrics-daemon-7qn5v" (UID: "9178e60e-1f54-461c-8a40-1b8c3300b64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:52.511362 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:52.511334    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:52.511424 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:52.511386    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret podName:078bb266-8297-48c9-a45b-2d47fb7cbd45 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:54.511370316 +0000 UTC m=+6.921893307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret") pod "global-pull-secret-syncer-xvkt5" (UID: "078bb266-8297-48c9-a45b-2d47fb7cbd45") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:52.612068 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.611732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdb9\" (UniqueName: \"kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9\") pod \"network-check-target-gcm28\" (UID: \"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1\") " pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:20:52.612068 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:52.611885    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:52.612068 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:52.611907    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:52.612068 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:52.611920    2559 projected.go:196] Error preparing data for projected volume kube-api-access-2pdb9 for pod openshift-network-diagnostics/network-check-target-gcm28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:52.612068 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:52.611977    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9 podName:4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:54.611959494 +0000 UTC m=+7.022482488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pdb9" (UniqueName: "kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9") pod "network-check-target-gcm28" (UID: "4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:52.830780 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.830699    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:50 +0000 UTC" deadline="2028-05-16 08:23:47.708762491 +0000 UTC"
Sep 19 08:20:52.830780 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.830738    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14520h2m54.878029174s"
Sep 19 08:20:52.932476 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.932448    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:52.932647 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:52.932577    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:20:52.933025 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.933004    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:52.933144 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:52.933121    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:20:52.933245 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:52.933204    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:20:52.933314 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:52.933294    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:20:53.129321 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:53.129227    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:53.647804 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:53.647771    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:54.529864 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:54.529832    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:54.530318 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:54.529908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:54.530318 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:54.530030    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:54.530318 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:54.530033    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:54.530318 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:54.530089    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret podName:078bb266-8297-48c9-a45b-2d47fb7cbd45 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:58.530071603 +0000 UTC m=+10.940594595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret") pod "global-pull-secret-syncer-xvkt5" (UID: "078bb266-8297-48c9-a45b-2d47fb7cbd45") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:54.530318 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:54.530107    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs podName:9178e60e-1f54-461c-8a40-1b8c3300b64c nodeName:}" failed. No retries permitted until 2026-09-19 08:20:58.530096913 +0000 UTC m=+10.940619895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs") pod "network-metrics-daemon-7qn5v" (UID: "9178e60e-1f54-461c-8a40-1b8c3300b64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:54.630827 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:54.630791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdb9\" (UniqueName: \"kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9\") pod \"network-check-target-gcm28\" (UID: \"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1\") " pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:20:54.631466 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:54.630986    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:54.631466 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:54.631013    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:54.631466 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:54.631027    2559 projected.go:196] Error preparing data for projected volume kube-api-access-2pdb9 for pod openshift-network-diagnostics/network-check-target-gcm28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:54.631466 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:54.631091    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9 podName:4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:58.631073786 +0000 UTC m=+11.041596770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pdb9" (UniqueName: "kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9") pod "network-check-target-gcm28" (UID: "4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:54.933984 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:54.933224    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:54.933984 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:54.933378    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:20:54.933984 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:54.933718    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:54.933984 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:54.933801    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:20:54.933984 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:54.933837    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:20:54.933984 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:54.933881    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:20:56.932531 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:56.932494    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:56.933011 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:56.932546    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:56.933011 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:56.932498    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:20:56.933011 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:56.932648    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:20:56.933011 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:56.932744    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:20:56.933011 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:56.932831    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:20:58.569925 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:58.569361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:58.569925 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:58.569425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:58.569925 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:58.569547    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:58.569925 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:58.569612    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs podName:9178e60e-1f54-461c-8a40-1b8c3300b64c nodeName:}" failed. No retries permitted until 2026-09-19 08:21:06.569592864 +0000 UTC m=+18.980115857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs") pod "network-metrics-daemon-7qn5v" (UID: "9178e60e-1f54-461c-8a40-1b8c3300b64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:58.570535 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:58.569956    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:58.570535 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:58.570027    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret podName:078bb266-8297-48c9-a45b-2d47fb7cbd45 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:06.570009739 +0000 UTC m=+18.980532731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret") pod "global-pull-secret-syncer-xvkt5" (UID: "078bb266-8297-48c9-a45b-2d47fb7cbd45") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:58.670055 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:58.670014    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdb9\" (UniqueName: \"kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9\") pod \"network-check-target-gcm28\" (UID: \"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1\") " pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:20:58.670242 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:58.670173    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:58.670242 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:58.670200    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:58.670242 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:58.670215    2559 projected.go:196] Error preparing data for projected volume kube-api-access-2pdb9 for pod openshift-network-diagnostics/network-check-target-gcm28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:58.670424 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:58.670315    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9 podName:4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:06.670295995 +0000 UTC m=+19.080818988 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pdb9" (UniqueName: "kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9") pod "network-check-target-gcm28" (UID: "4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:58.932479 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:58.932393    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:20:58.932655 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:58.932493    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:20:58.932655 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:58.932508    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:20:58.932655 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:20:58.932527    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:20:58.932655 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:58.932628    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:20:58.932790 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:20:58.932722    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:00.932201 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:00.932164    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:00.932688 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:00.932166    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:00.932688 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:00.932306    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:00.932688 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:00.932164    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:00.932688 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:00.932436    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:00.932688 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:00.932557    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:02.932368 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:02.932329    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:02.932811 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:02.932329    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:02.932811 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:02.932451    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:02.932811 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:02.932461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:02.932811 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:02.932564    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:02.932811 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:02.932653    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:04.931917 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:04.931875    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:04.931917 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:04.931891    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:04.931917 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:04.931891    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:04.932512 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:04.931983    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:04.932512 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:04.932081    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:04.932512 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:04.932130    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:06.624970 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:06.624931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:06.625525 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:06.625004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:06.625525 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:06.625113    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:06.625525 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:06.625191    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs podName:9178e60e-1f54-461c-8a40-1b8c3300b64c nodeName:}" failed. No retries permitted until 2026-09-19 08:21:22.625176171 +0000 UTC m=+35.035699155 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs") pod "network-metrics-daemon-7qn5v" (UID: "9178e60e-1f54-461c-8a40-1b8c3300b64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:06.625525 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:06.625113    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:06.625525 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:06.625287    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret podName:078bb266-8297-48c9-a45b-2d47fb7cbd45 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:22.625265998 +0000 UTC m=+35.035788989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret") pod "global-pull-secret-syncer-xvkt5" (UID: "078bb266-8297-48c9-a45b-2d47fb7cbd45") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:06.725897 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:06.725859    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdb9\" (UniqueName: \"kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9\") pod \"network-check-target-gcm28\" (UID: \"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1\") " pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:06.726089 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:06.726064    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:21:06.726139 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:06.726100    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:21:06.726139 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:06.726114    2559 projected.go:196] Error preparing data for projected volume kube-api-access-2pdb9 for pod openshift-network-diagnostics/network-check-target-gcm28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:06.726229 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:06.726182    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9 podName:4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:22.726162248 +0000 UTC m=+35.136685245 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pdb9" (UniqueName: "kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9") pod "network-check-target-gcm28" (UID: "4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:06.932571 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:06.932479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:06.932744 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:06.932479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:06.932744 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:06.932622    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:06.932744 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:06.932477    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:06.932744 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:06.932690    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:06.932999 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:06.932747    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:08.932622 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:08.932585    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:08.933180 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:08.932625    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:08.933180 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:08.932632    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:08.933180 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:08.932713    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:08.933180 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:08.932807    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:08.933180 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:08.932899    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:10.208470 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:10.207930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tnb9c" event={"ID":"bca89dd8-76bb-4909-be5e-91f7f7fa40bc","Type":"ContainerStarted","Data":"660d2c8909f88ed7059baa18d89c205c024f0103cb2c810f4b2a9a3e5c5ae2b1"}
Sep 19 08:21:10.216439 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:10.216402    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pv57b" event={"ID":"007b6a79-68a9-4bf6-8d49-897b5151c149","Type":"ContainerStarted","Data":"93237beb6ac8920c7473a21041212512e2cea307c6818915075309f57cc3239f"}
Sep 19 08:21:10.225903 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:10.225877    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal" event={"ID":"3a7baa0c5fc4f5272bafad3e803d0c95","Type":"ContainerStarted","Data":"9f43b5e8374f55ae32b4e840b368ca04596acaef6f03a26d44488bc602a70d9c"}
Sep 19 08:21:10.237504 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:10.237434    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tnb9c" podStartSLOduration=1.55021591 podStartE2EDuration="20.237413375s" totalImagesPullingTime="18.687197465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:10.237302947 +0000 UTC m=+22.647825950" watchObservedRunningTime="2026-09-19 08:21:10.237413375 +0000 UTC m=+22.647936375"
Sep 19 08:21:10.275388 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:10.274597    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pv57b" podStartSLOduration=1.556356728 podStartE2EDuration="20.274577389s" totalImagesPullingTime="18.718220661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:10.27455704 +0000 UTC m=+22.685080041" watchObservedRunningTime="2026-09-19 08:21:10.274577389 +0000 UTC m=+22.685100393"
Sep 19 08:21:10.275519 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:10.275412    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-26.ec2.internal" podStartSLOduration=20.275398572 podStartE2EDuration="20.275398572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:10.254671158 +0000 UTC m=+22.665194153" watchObservedRunningTime="2026-09-19 08:21:10.275398572 +0000 UTC m=+22.685921572"
Sep 19 08:21:10.932646 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:10.932406    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:10.932866 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:10.932454    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:10.932866 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:10.932723    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:10.932866 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:10.932827    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:10.932866 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:10.932460    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:10.933064 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:10.932912    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:11.230546 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.230509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lcmrp" event={"ID":"2d618e83-8b27-4401-94d2-1feaa04c634f","Type":"ContainerStarted","Data":"5faa15c3a239e11656d73d723fd84170ae611640474e384e5e895f63e69f3a1d"}
Sep 19 08:21:11.232413 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.232383    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h46xh" event={"ID":"955095d1-46e8-49b9-a667-a40d58fa50ef","Type":"ContainerStarted","Data":"8b951244dc8df194009251a420b399e39536baadff1818b85a469bfb3f8cd7b0"}
Sep 19 08:21:11.234303 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.234272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns" event={"ID":"d3ed0f54-9d5e-4fd0-a672-761615ba1eba","Type":"ContainerStarted","Data":"2a27c217a67a72d818efce724b0cf35566a65d97cf6aa5806e5fcceee1c8e9af"}
Sep 19 08:21:11.236074 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.236043    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xn4mf" event={"ID":"af211ba6-f358-4565-a7dd-37eb2337ea07","Type":"ContainerStarted","Data":"f2a088da2e5fa10946c7cb1e3c44e256445ed01f492ce713bc74f4d6584eaae2"}
Sep 19 08:21:11.239666 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.239639    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5" event={"ID":"338a6f68-5bed-457b-b1f2-23f4e68922d6","Type":"ContainerStarted","Data":"935c7e1a663cff85bf9528ab7eb300f86ebc04bf8e6adbd1bcea883ab56c3128"}
Sep 19 08:21:11.239797 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.239710    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5" event={"ID":"338a6f68-5bed-457b-b1f2-23f4e68922d6","Type":"ContainerStarted","Data":"f833a0682e6a72c0dcba777858ccdb0826b766c65941da9b9f883a7019802960"}
Sep 19 08:21:11.239797 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.239758    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5" event={"ID":"338a6f68-5bed-457b-b1f2-23f4e68922d6","Type":"ContainerStarted","Data":"1435404da27a2c3b94d1444aa433eae5e961998eb492fd14c706b8cff0662c03"}
Sep 19 08:21:11.239906 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.239806    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5" event={"ID":"338a6f68-5bed-457b-b1f2-23f4e68922d6","Type":"ContainerStarted","Data":"df249fddbb1589476532ea6ea0ca981bf52475c114edc7bc87b496c27effa5e2"}
Sep 19 08:21:11.239906 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.239847    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5" event={"ID":"338a6f68-5bed-457b-b1f2-23f4e68922d6","Type":"ContainerStarted","Data":"5874bfa5fcdc2cf0ae66a13b9a5384983f34973b388191f005b935fdb2c6a3df"}
Sep 19 08:21:11.239906 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.239884    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5" event={"ID":"338a6f68-5bed-457b-b1f2-23f4e68922d6","Type":"ContainerStarted","Data":"8e8919c7b55dfb2e63b4a174b2258e62b6712de05a9177194159e8596ab6c65b"}
Sep 19 08:21:11.241231 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.241204    2559 generic.go:356] "Generic (PLEG): container finished" podID="768fc03a5f9fae3a346f862bcd558866" containerID="99f946b5a819475f60561eeee1386967932f9efdbe836235f992d53ca8dca146" exitCode=0
Sep 19 08:21:11.241360 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.241289    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal" event={"ID":"768fc03a5f9fae3a346f862bcd558866","Type":"ContainerDied","Data":"99f946b5a819475f60561eeee1386967932f9efdbe836235f992d53ca8dca146"}
Sep 19 08:21:11.245033 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.245006    2559 generic.go:356] "Generic (PLEG): container finished" podID="9fb5bd3b-10c8-41fb-b8b1-d14846935295" containerID="eeb88c44dc7891d204ad5fb1a004418e9a90082050473fce4e0928d67f32acf1" exitCode=0
Sep 19 08:21:11.245125 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.245093    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2x69" event={"ID":"9fb5bd3b-10c8-41fb-b8b1-d14846935295","Type":"ContainerDied","Data":"eeb88c44dc7891d204ad5fb1a004418e9a90082050473fce4e0928d67f32acf1"}
Sep 19 08:21:11.246887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.246739    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lcmrp" podStartSLOduration=2.662936762 podStartE2EDuration="21.246727672s" totalImagesPullingTime="18.58379091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.246441184 +0000 UTC m=+23.656964184" watchObservedRunningTime="2026-09-19 08:21:11.246727672 +0000 UTC m=+23.657250671"
Sep 19 08:21:11.267859 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.267811    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xn4mf" podStartSLOduration=2.636623068 podStartE2EDuration="21.267797961s" totalImagesPullingTime="18.631174893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.267395808 +0000 UTC m=+23.677918812" watchObservedRunningTime="2026-09-19 08:21:11.267797961 +0000 UTC m=+23.678320960"
Sep 19 08:21:11.285248 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.285203    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h46xh" podStartSLOduration=2.6267198609999998 podStartE2EDuration="21.285188755s" totalImagesPullingTime="18.658468894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.285029321 +0000 UTC m=+23.695552321" watchObservedRunningTime="2026-09-19 08:21:11.285188755 +0000 UTC m=+23.695711769"
Sep 19 08:21:11.467448 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.467419    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:11.545543 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.545470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:11.734748 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.734718    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 19 08:21:11.881975 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.881759    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T08:21:11.734739365Z","UUID":"c80b4fe8-8791-4cca-9d47-1e4d7bfc2c7d","Handler":null,"Name":"","Endpoint":""}
Sep 19 08:21:11.883402 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.883370    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 19 08:21:11.883561 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:11.883412    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 19 08:21:12.245593 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.245506    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:21:12.246200 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.246180    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:21:12.250189 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.250154    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns" event={"ID":"d3ed0f54-9d5e-4fd0-a672-761615ba1eba","Type":"ContainerStarted","Data":"aca28cf65f6fd5b1fa18119b0681f4bf6ce606e5c3cbe478898d921c66d3bfbf"}
Sep 19 08:21:12.252581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.252550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wnn76" event={"ID":"8ac0106e-5a03-4fcb-a733-5fd551497dc0","Type":"ContainerStarted","Data":"2b57707e5db4e3dad2d8b9547b1be45131fd86a257cd7c99ca516c5468c868f5"}
Sep 19 08:21:12.255117 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.255093    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal" event={"ID":"768fc03a5f9fae3a346f862bcd558866","Type":"ContainerStarted","Data":"bc4f05868c40a74ec1abeae34199b054acd303cc180f5ccb802481389cce5fbc"}
Sep 19 08:21:12.279136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.279078    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-26.ec2.internal" podStartSLOduration=22.279060742 podStartE2EDuration="22.279060742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:12.278858068 +0000 UTC m=+24.689381069" watchObservedRunningTime="2026-09-19 08:21:12.279060742 +0000 UTC m=+24.689583742"
Sep 19 08:21:12.293032 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.292973    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wnn76" podStartSLOduration=3.6433779619999997 podStartE2EDuration="22.29295469s" totalImagesPullingTime="18.649576728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:12.29275711 +0000 UTC m=+24.703280113" watchObservedRunningTime="2026-09-19 08:21:12.29295469 +0000 UTC m=+24.703477692"
Sep 19 08:21:12.413997 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.413967    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:12.496848 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.496754    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:12.894497 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.894440    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:12.932683 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.932652    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:12.932830 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.932685    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:12.932830 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:12.932695    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:12.932830 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:12.932791    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:12.932999 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:12.932908    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:12.933074 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:12.933009    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:13.024838 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:13.024799    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:13.097160 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:13.097068    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:13.262017 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:13.261979    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5" event={"ID":"338a6f68-5bed-457b-b1f2-23f4e68922d6","Type":"ContainerStarted","Data":"7cee62c1582d054b637469bd6ae2dea955dfb7f3d652f2b37a304716b04b2562"}
Sep 19 08:21:13.264712 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:13.264678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns" event={"ID":"d3ed0f54-9d5e-4fd0-a672-761615ba1eba","Type":"ContainerStarted","Data":"e676c3b0bd7ed8740b30cd8640d058f2474cd7f3d3bad0c6ce0eb4f3e4a72eb5"}
Sep 19 08:21:13.264712 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:13.264696    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 08:21:13.284670 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:13.284612    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdlns" podStartSLOduration=2.466982153 podStartE2EDuration="23.284590777s" totalImagesPullingTime="20.817608624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:13.283667399 +0000 UTC m=+25.694190399" watchObservedRunningTime="2026-09-19 08:21:13.284590777 +0000 UTC m=+25.695113760"
Sep 19 08:21:13.948647 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:13.948453    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:14.049429 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:14.049398    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:14.283123 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:14.283096    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:21:14.283598 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:14.283210    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 08:21:14.283846 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:14.283826    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h46xh"
Sep 19 08:21:14.346664 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:14.346635    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:14.567533 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:14.567453    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:14.672150 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:14.672114    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:14.932948 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:14.932853    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:14.932948 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:14.932902    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:14.932948 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:14.932902    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:14.933227 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:14.932975    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:14.933227 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:14.933109    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:14.933227 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:14.933205    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:14.981813 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:14.981782    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:15.051672 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:15.051638    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:15.628098 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:15.627868    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:15.775732 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:15.775647    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:16.278023 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:16.277986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5" event={"ID":"338a6f68-5bed-457b-b1f2-23f4e68922d6","Type":"ContainerStarted","Data":"1d7c7298d269b1cb591f365b71cd2d0ac8ac2b1c24acf93cfca0734eb9a6b1b3"}
Sep 19 08:21:16.278389 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:16.278359    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:21:16.280127 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:16.280101    2559 generic.go:356] "Generic (PLEG): container finished" podID="9fb5bd3b-10c8-41fb-b8b1-d14846935295" containerID="651c424d0f662a8995e8a7a267aa4e1bc1d8fd2621be6caa8a8f0094b63b4377" exitCode=0
Sep 19 08:21:16.280230 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:16.280137    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2x69" event={"ID":"9fb5bd3b-10c8-41fb-b8b1-d14846935295","Type":"ContainerDied","Data":"651c424d0f662a8995e8a7a267aa4e1bc1d8fd2621be6caa8a8f0094b63b4377"}
Sep 19 08:21:16.294699 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:16.294681    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:21:16.353051 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:16.353000    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5" podStartSLOduration=7.37682556 podStartE2EDuration="26.352983746s" totalImagesPullingTime="18.976158186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:16.320438034 +0000 UTC m=+28.730961033" watchObservedRunningTime="2026-09-19 08:21:16.352983746 +0000 UTC m=+28.763506746"
Sep 19 08:21:16.932749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:16.932717    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:16.933105 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:16.932841    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:16.933105 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:16.932856    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:16.933105 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:16.932963    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:16.933105 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:16.933026    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:16.933105 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:16.933088    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:17.212059 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.211971    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:17.262393 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.262364    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xvkt5"]
Sep 19 08:21:17.263436 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.263406    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gcm28"]
Sep 19 08:21:17.264048 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.264021    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qn5v"]
Sep 19 08:21:17.282472 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.282443    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:17.285588 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.285556    2559 generic.go:356] "Generic (PLEG): container finished" podID="9fb5bd3b-10c8-41fb-b8b1-d14846935295" containerID="8021ba2d204a80cf53cfb79cc27879af23bc66f4eda5d64a97fd6ec31e3ac443" exitCode=0
Sep 19 08:21:17.285588 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.285581    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2x69" event={"ID":"9fb5bd3b-10c8-41fb-b8b1-d14846935295","Type":"ContainerDied","Data":"8021ba2d204a80cf53cfb79cc27879af23bc66f4eda5d64a97fd6ec31e3ac443"}
Sep 19 08:21:17.285764 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.285722    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:17.285764 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.285741    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:17.285876 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:17.285855    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:17.285980 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:17.285957    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:17.286085 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.286071    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:17.287093 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:17.286498    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:17.287093 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.286531    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:21:17.287093 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.286593    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:21:17.304437 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:17.304414    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:21:18.290786 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:18.290750    2559 generic.go:356] "Generic (PLEG): container finished" podID="9fb5bd3b-10c8-41fb-b8b1-d14846935295" containerID="de902a4ad93835d9f3f1d7b1cc8b0eb46779ced7608f7509caa2d1619d5bc3f2" exitCode=0
Sep 19 08:21:18.291240 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:18.290830    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2x69" event={"ID":"9fb5bd3b-10c8-41fb-b8b1-d14846935295","Type":"ContainerDied","Data":"de902a4ad93835d9f3f1d7b1cc8b0eb46779ced7608f7509caa2d1619d5bc3f2"}
Sep 19 08:21:18.932722 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:18.932684    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:18.932883 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:18.932684    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:18.932883 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:18.932836    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:18.932999 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:18.932937    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:18.932999 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:18.932684    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:18.933098 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:18.933054    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:20.931939 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:20.931904    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:20.932417 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:20.931903    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:20.932417 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:20.932040    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:20.932417 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:20.931915    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:20.932417 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:20.932127    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:20.932417 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:20.932183    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:22.650976 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:22.650937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:22.651398 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:22.651019    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:22.651398 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:22.651086    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:22.651398 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:22.651134    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:22.651398 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:22.651150    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret podName:078bb266-8297-48c9-a45b-2d47fb7cbd45 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:54.651133064 +0000 UTC m=+67.061656042 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret") pod "global-pull-secret-syncer-xvkt5" (UID: "078bb266-8297-48c9-a45b-2d47fb7cbd45") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:22.651398 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:22.651183    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs podName:9178e60e-1f54-461c-8a40-1b8c3300b64c nodeName:}" failed. No retries permitted until 2026-09-19 08:21:54.651167299 +0000 UTC m=+67.061690295 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs") pod "network-metrics-daemon-7qn5v" (UID: "9178e60e-1f54-461c-8a40-1b8c3300b64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:22.752220 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:22.752193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdb9\" (UniqueName: \"kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9\") pod \"network-check-target-gcm28\" (UID: \"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1\") " pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:22.752368 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:22.752353    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:21:22.752410 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:22.752374    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:21:22.752410 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:22.752383    2559 projected.go:196] Error preparing data for projected volume kube-api-access-2pdb9 for pod openshift-network-diagnostics/network-check-target-gcm28: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:22.752475 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:22.752435    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9 podName:4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:54.752419709 +0000 UTC m=+67.162942711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2pdb9" (UniqueName: "kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9") pod "network-check-target-gcm28" (UID: "4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:22.932576 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:22.932509    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:22.932734 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:22.932605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:22.932734 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:22.932608    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gcm28" podUID="4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1"
Sep 19 08:21:22.932734 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:22.932614    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:22.932734 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:22.932691    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xvkt5" podUID="078bb266-8297-48c9-a45b-2d47fb7cbd45"
Sep 19 08:21:22.932856 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:22.932763    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qn5v" podUID="9178e60e-1f54-461c-8a40-1b8c3300b64c"
Sep 19 08:21:23.308103 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.308073    2559 generic.go:356] "Generic (PLEG): container finished" podID="9fb5bd3b-10c8-41fb-b8b1-d14846935295" containerID="7ed66423603ab293402e73ee6a1c0e0b341b64bc67106a6e36055bf91dd466f7" exitCode=0
Sep 19 08:21:23.308375 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.308113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2x69" event={"ID":"9fb5bd3b-10c8-41fb-b8b1-d14846935295","Type":"ContainerDied","Data":"7ed66423603ab293402e73ee6a1c0e0b341b64bc67106a6e36055bf91dd466f7"}
Sep 19 08:21:23.730921 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.730892    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-26.ec2.internal" event="NodeReady"
Sep 19 08:21:23.731507 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.731032    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 19 08:21:23.826073 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.825991    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-45sdz"]
Sep 19 08:21:23.846530 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.846504    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-svmsl"]
Sep 19 08:21:23.846675 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.846632    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-45sdz"
Sep 19 08:21:23.851318 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.851291    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 19 08:21:23.851467 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.851384    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 19 08:21:23.851601 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.851585    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vbr8d\""
Sep 19 08:21:23.851653 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.851591    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 19 08:21:23.866665 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.866644    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-45sdz"]
Sep 19 08:21:23.866665 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.866672    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-svmsl"]
Sep 19 08:21:23.866838 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.866767    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:23.869991 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.869965    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 19 08:21:23.870116 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.869965    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g2bvg\""
Sep 19 08:21:23.870203 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.870189    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 19 08:21:23.959117 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.959085    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:23.959117 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.959114    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8098f93f-f309-4e66-a1a5-fe28c2b8a5d8-cert\") pod \"ingress-canary-45sdz\" (UID: \"8098f93f-f309-4e66-a1a5-fe28c2b8a5d8\") " pod="openshift-ingress-canary/ingress-canary-45sdz"
Sep 19 08:21:23.959342 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.959133    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6663b01e-cdec-4748-a1b3-7fd84903131d-config-volume\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:23.959342 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.959231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6663b01e-cdec-4748-a1b3-7fd84903131d-tmp-dir\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:23.959342 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.959293    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbkrs\" (UniqueName: \"kubernetes.io/projected/8098f93f-f309-4e66-a1a5-fe28c2b8a5d8-kube-api-access-qbkrs\") pod \"ingress-canary-45sdz\" (UID: \"8098f93f-f309-4e66-a1a5-fe28c2b8a5d8\") " pod="openshift-ingress-canary/ingress-canary-45sdz"
Sep 19 08:21:23.959342 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:23.959328    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkflt\" (UniqueName: \"kubernetes.io/projected/6663b01e-cdec-4748-a1b3-7fd84903131d-kube-api-access-qkflt\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:24.041005 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.040955    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:24.059959 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.059931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6663b01e-cdec-4748-a1b3-7fd84903131d-tmp-dir\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:24.060074 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.059965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbkrs\" (UniqueName: \"kubernetes.io/projected/8098f93f-f309-4e66-a1a5-fe28c2b8a5d8-kube-api-access-qbkrs\") pod \"ingress-canary-45sdz\" (UID: \"8098f93f-f309-4e66-a1a5-fe28c2b8a5d8\") " pod="openshift-ingress-canary/ingress-canary-45sdz"
Sep 19 08:21:24.060074 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.060004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkflt\" (UniqueName: \"kubernetes.io/projected/6663b01e-cdec-4748-a1b3-7fd84903131d-kube-api-access-qkflt\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:24.060074 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.060056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:24.060274 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.060081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8098f93f-f309-4e66-a1a5-fe28c2b8a5d8-cert\") pod \"ingress-canary-45sdz\" (UID: \"8098f93f-f309-4e66-a1a5-fe28c2b8a5d8\") " pod="openshift-ingress-canary/ingress-canary-45sdz"
Sep 19 08:21:24.060274 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.060106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6663b01e-cdec-4748-a1b3-7fd84903131d-config-volume\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:24.060274 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:24.060189    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:24.060274 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:24.060245    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls podName:6663b01e-cdec-4748-a1b3-7fd84903131d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:24.56022822 +0000 UTC m=+36.970751217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls") pod "dns-default-svmsl" (UID: "6663b01e-cdec-4748-a1b3-7fd84903131d") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:24.060468 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.060394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6663b01e-cdec-4748-a1b3-7fd84903131d-tmp-dir\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:24.060649 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.060631    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6663b01e-cdec-4748-a1b3-7fd84903131d-config-volume\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:24.064583 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.064560    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8098f93f-f309-4e66-a1a5-fe28c2b8a5d8-cert\") pod \"ingress-canary-45sdz\" (UID: \"8098f93f-f309-4e66-a1a5-fe28c2b8a5d8\") " pod="openshift-ingress-canary/ingress-canary-45sdz"
Sep 19 08:21:24.071283 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.070943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbkrs\" (UniqueName: \"kubernetes.io/projected/8098f93f-f309-4e66-a1a5-fe28c2b8a5d8-kube-api-access-qbkrs\") pod \"ingress-canary-45sdz\" (UID: \"8098f93f-f309-4e66-a1a5-fe28c2b8a5d8\") " pod="openshift-ingress-canary/ingress-canary-45sdz"
Sep 19 08:21:24.071740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.071719    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkflt\" (UniqueName: \"kubernetes.io/projected/6663b01e-cdec-4748-a1b3-7fd84903131d-kube-api-access-qkflt\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:24.143934 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.143812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:24.155636 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.155613    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-45sdz"
Sep 19 08:21:24.314135 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.314100    2559 generic.go:356] "Generic (PLEG): container finished" podID="9fb5bd3b-10c8-41fb-b8b1-d14846935295" containerID="a9a7aeb7aeb75d7825e22a46a68d4483eb3dd7c6c95a973ef1a164cef1e61dc1" exitCode=0
Sep 19 08:21:24.314316 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.314163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2x69" event={"ID":"9fb5bd3b-10c8-41fb-b8b1-d14846935295","Type":"ContainerDied","Data":"a9a7aeb7aeb75d7825e22a46a68d4483eb3dd7c6c95a973ef1a164cef1e61dc1"}
Sep 19 08:21:24.505096 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.504828    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-45sdz"]
Sep 19 08:21:24.508889 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:21:24.508856    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8098f93f_f309_4e66_a1a5_fe28c2b8a5d8.slice/crio-6201200aea13309dcb8a30be1303a85c9d30dcd9adbf8a61a4d8cf23dd1a7494 WatchSource:0}: Error finding container 6201200aea13309dcb8a30be1303a85c9d30dcd9adbf8a61a4d8cf23dd1a7494: Status 404 returned error can't find the container with id 6201200aea13309dcb8a30be1303a85c9d30dcd9adbf8a61a4d8cf23dd1a7494
Sep 19 08:21:24.564268 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.564222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:24.564417 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:24.564374    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:24.564457 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:24.564436    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls podName:6663b01e-cdec-4748-a1b3-7fd84903131d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:25.564421377 +0000 UTC m=+37.974944354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls") pod "dns-default-svmsl" (UID: "6663b01e-cdec-4748-a1b3-7fd84903131d") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:24.932578 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.932543    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:24.933360 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.932581    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:24.933360 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.932641    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:24.935022 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.934970    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:24.936019 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.935988    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:24.936151 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.936083    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 08:21:24.936151 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.936132    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 08:21:24.936812 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.936789    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47kpw\""
Sep 19 08:21:24.936938 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.936826    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c9bjw\""
Sep 19 08:21:24.964101 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:24.964068    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:25.064185 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:25.064148    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:25.321405 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:25.321370    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2x69" event={"ID":"9fb5bd3b-10c8-41fb-b8b1-d14846935295","Type":"ContainerStarted","Data":"f0703af58f5a686dd5a1abde6efe4bfff32094d8d8d6c4bcf45705ea5b885726"}
Sep 19 08:21:25.322695 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:25.322664    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-45sdz" event={"ID":"8098f93f-f309-4e66-a1a5-fe28c2b8a5d8","Type":"ContainerStarted","Data":"6201200aea13309dcb8a30be1303a85c9d30dcd9adbf8a61a4d8cf23dd1a7494"}
Sep 19 08:21:25.359413 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:25.359349    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g2x69" podStartSLOduration=7.054143232 podStartE2EDuration="35.359327011s" totalImagesPullingTime="28.305183779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:25.35815054 +0000 UTC m=+37.768673540" watchObservedRunningTime="2026-09-19 08:21:25.359327011 +0000 UTC m=+37.769850012"
Sep 19 08:21:25.498535 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:25.498499    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:25.573028 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:25.572948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:25.573188 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:25.573092    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:25.573188 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:25.573161    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls podName:6663b01e-cdec-4748-a1b3-7fd84903131d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:27.573143008 +0000 UTC m=+39.983665993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls") pod "dns-default-svmsl" (UID: "6663b01e-cdec-4748-a1b3-7fd84903131d") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:25.598923 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:25.598884    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:26.924930 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:26.924884    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:27.025459 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:27.025422    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:27.331718 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:27.331683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-45sdz" event={"ID":"8098f93f-f309-4e66-a1a5-fe28c2b8a5d8","Type":"ContainerStarted","Data":"907a7de74d76c103d283955b34ee43a77f83b51b5f5e54a2248749b6e3788acf"}
Sep 19 08:21:27.350810 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:27.350721    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-45sdz" podStartSLOduration=2.006343854 podStartE2EDuration="4.35070652s" totalImagesPullingTime="2.344362666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:27.35036648 +0000 UTC m=+39.760889480" watchObservedRunningTime="2026-09-19 08:21:27.35070652 +0000 UTC m=+39.761229553"
Sep 19 08:21:27.590438 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:27.590396    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:27.590618 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:27.590535    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:27.590618 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:27.590602    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls podName:6663b01e-cdec-4748-a1b3-7fd84903131d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:31.590586053 +0000 UTC m=+44.001109031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls") pod "dns-default-svmsl" (UID: "6663b01e-cdec-4748-a1b3-7fd84903131d") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:27.896304 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:27.896107    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:27.971161 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:27.971127    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:30.020369 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:30.020334    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:30.081007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:30.080975    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:31.618440 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:31.618397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:31.618846 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:31.618551    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:31.618846 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:31.618618    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls podName:6663b01e-cdec-4748-a1b3-7fd84903131d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.618601737 +0000 UTC m=+52.029124716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls") pod "dns-default-svmsl" (UID: "6663b01e-cdec-4748-a1b3-7fd84903131d") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:35.188353 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:35.188321    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:35.256065 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:35.256036    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:38.150443 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.150412    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gfznx"]
Sep 19 08:21:38.188371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.188344    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-hhscx"]
Sep 19 08:21:38.188529 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.188464    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gfznx"
Sep 19 08:21:38.190842 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.190818    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-msnr8\""
Sep 19 08:21:38.191031 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.191018    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:21:38.191838 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.191821    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 19 08:21:38.202086 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.202069    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gfznx"]
Sep 19 08:21:38.202159 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.202096    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-hhscx"]
Sep 19 08:21:38.202193 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.202165    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.205191 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.205173    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 19 08:21:38.205313 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.205225    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-kqrth\""
Sep 19 08:21:38.205418 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.205401    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 08:21:38.206492 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.206475    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 19 08:21:38.206492 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.206483    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 08:21:38.213996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.213976    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 19 08:21:38.255565 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.255532    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"]
Sep 19 08:21:38.274018 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.273992    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"]
Sep 19 08:21:38.274159 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.274083    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"
Sep 19 08:21:38.280164 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.280141    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:21:38.280313 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.280189    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 19 08:21:38.280386 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.280313    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4z5pd\""
Sep 19 08:21:38.280976 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.280945    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 19 08:21:38.281096 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.281080    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 19 08:21:38.288129 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.288111    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"]
Sep 19 08:21:38.288208 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.288159    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.292182 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.292166    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 19 08:21:38.292388 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.292376    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v4wbb\""
Sep 19 08:21:38.292605 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.292587    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 19 08:21:38.292651 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.292621    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 19 08:21:38.320572 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.320551    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"]
Sep 19 08:21:38.326848 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.326834    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 19 08:21:38.365590 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.365561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-snapshots\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.365590 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.365589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ft8\" (UniqueName: \"kubernetes.io/projected/af453c92-08c7-44c0-9e10-52cf98b6b7f4-kube-api-access-f9ft8\") pod \"kube-storage-version-migrator-operator-55889f7b47-k7g64\" (UID: \"af453c92-08c7-44c0-9e10-52cf98b6b7f4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"
Sep 19 08:21:38.365777 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.365611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af453c92-08c7-44c0-9e10-52cf98b6b7f4-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-k7g64\" (UID: \"af453c92-08c7-44c0-9e10-52cf98b6b7f4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"
Sep 19 08:21:38.365777 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.365661    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af453c92-08c7-44c0-9e10-52cf98b6b7f4-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-k7g64\" (UID: \"af453c92-08c7-44c0-9e10-52cf98b6b7f4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"
Sep 19 08:21:38.365777 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.365692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.365777 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.365710    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-serving-cert\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.365777 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.365725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m2wd\" (UniqueName: \"kubernetes.io/projected/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-kube-api-access-7m2wd\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.365777 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.365759    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zwm4\" (UniqueName: \"kubernetes.io/projected/cfe03657-f8db-4e34-90f3-e6762a4f8753-kube-api-access-2zwm4\") pod \"volume-data-source-validator-5c4db8bdf9-gfznx\" (UID: \"cfe03657-f8db-4e34-90f3-e6762a4f8753\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gfznx"
Sep 19 08:21:38.366008 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.365835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-tmp\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.366008 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.365858    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.466362 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af453c92-08c7-44c0-9e10-52cf98b6b7f4-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-k7g64\" (UID: \"af453c92-08c7-44c0-9e10-52cf98b6b7f4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"
Sep 19 08:21:38.466362 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-registry-certificates\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.466362 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466332    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc5h7\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-kube-api-access-mc5h7\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.466612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466371    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af453c92-08c7-44c0-9e10-52cf98b6b7f4-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-k7g64\" (UID: \"af453c92-08c7-44c0-9e10-52cf98b6b7f4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"
Sep 19 08:21:38.466612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-bound-sa-token\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.466612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466457    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.466612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466504    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-serving-cert\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.466612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m2wd\" (UniqueName: \"kubernetes.io/projected/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-kube-api-access-7m2wd\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.466612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466563    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-trusted-ca\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.466904 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwm4\" (UniqueName: \"kubernetes.io/projected/cfe03657-f8db-4e34-90f3-e6762a4f8753-kube-api-access-2zwm4\") pod \"volume-data-source-validator-5c4db8bdf9-gfznx\" (UID: \"cfe03657-f8db-4e34-90f3-e6762a4f8753\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gfznx"
Sep 19 08:21:38.466904 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466685    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-image-registry-private-configuration\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.466904 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466725    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-tmp\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.466904 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.466904 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466790    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.466904 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5bace44-66ba-4d16-8634-a518af242b6c-ca-trust-extracted\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.467170 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466912    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-snapshots\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.467170 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.466961    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ft8\" (UniqueName: \"kubernetes.io/projected/af453c92-08c7-44c0-9e10-52cf98b6b7f4-kube-api-access-f9ft8\") pod \"kube-storage-version-migrator-operator-55889f7b47-k7g64\" (UID: \"af453c92-08c7-44c0-9e10-52cf98b6b7f4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"
Sep 19 08:21:38.467170 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.467002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-installation-pull-secrets\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.467170 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.467036    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/af453c92-08c7-44c0-9e10-52cf98b6b7f4-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-k7g64\" (UID: \"af453c92-08c7-44c0-9e10-52cf98b6b7f4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"
Sep 19 08:21:38.467361 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.467227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-tmp\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.467612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.467595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-snapshots\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.467693 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.467672    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.467991 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.467976    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.471353 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.471324    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/af453c92-08c7-44c0-9e10-52cf98b6b7f4-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-k7g64\" (UID: \"af453c92-08c7-44c0-9e10-52cf98b6b7f4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"
Sep 19 08:21:38.471463 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.471385    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-serving-cert\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.477192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.477168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m2wd\" (UniqueName: \"kubernetes.io/projected/60dc08bc-8f96-458d-a5a7-c09534ca4ef5-kube-api-access-7m2wd\") pod \"insights-operator-6f9c644dd9-hhscx\" (UID: \"60dc08bc-8f96-458d-a5a7-c09534ca4ef5\") " pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.477336 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.477247    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ft8\" (UniqueName: \"kubernetes.io/projected/af453c92-08c7-44c0-9e10-52cf98b6b7f4-kube-api-access-f9ft8\") pod \"kube-storage-version-migrator-operator-55889f7b47-k7g64\" (UID: \"af453c92-08c7-44c0-9e10-52cf98b6b7f4\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"
Sep 19 08:21:38.477602 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.477582    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zwm4\" (UniqueName: \"kubernetes.io/projected/cfe03657-f8db-4e34-90f3-e6762a4f8753-kube-api-access-2zwm4\") pod \"volume-data-source-validator-5c4db8bdf9-gfznx\" (UID: \"cfe03657-f8db-4e34-90f3-e6762a4f8753\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gfznx"
Sep 19 08:21:38.491322 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.491287    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:38.497233 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.497217    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gfznx"
Sep 19 08:21:38.511384 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.511367    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-hhscx"
Sep 19 08:21:38.567625 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.567572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-trusted-ca\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.567805 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.567663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-image-registry-private-configuration\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.567805 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.567721    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.567805 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.567763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5bace44-66ba-4d16-8634-a518af242b6c-ca-trust-extracted\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.567962 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.567875    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-installation-pull-secrets\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.567962 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.567921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-registry-certificates\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.567962 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.567953    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc5h7\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-kube-api-access-mc5h7\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.568098 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.568004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-bound-sa-token\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.568567 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.568512    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5bace44-66ba-4d16-8634-a518af242b6c-ca-trust-extracted\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.568992 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:38.568731    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:38.569102 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:38.569000    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcfbdb9c6-wc2zp: secret "image-registry-tls" not found
Sep 19 08:21:38.569102 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:38.569085    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls podName:b5bace44-66ba-4d16-8634-a518af242b6c nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.069063722 +0000 UTC m=+51.479586719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls") pod "image-registry-bcfbdb9c6-wc2zp" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c") : secret "image-registry-tls" not found
Sep 19 08:21:38.569222 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.568918    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-trusted-ca\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.569576 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.569556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-registry-certificates\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.571516 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.571487    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-image-registry-private-configuration\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.572165 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.572119    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-installation-pull-secrets\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.577371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.577189    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc5h7\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-kube-api-access-mc5h7\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.577517 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.577486    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-bound-sa-token\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:38.584110 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.584084    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"
Sep 19 08:21:38.591881 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.591844    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:38.798054 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.798021    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gfznx"]
Sep 19 08:21:38.801229 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:21:38.801201    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe03657_f8db_4e34_90f3_e6762a4f8753.slice/crio-8970e6d2934cec4a43d044e03239fb6b9b1b0e64a0c9d1be2837e0c3df4e86c6 WatchSource:0}: Error finding container 8970e6d2934cec4a43d044e03239fb6b9b1b0e64a0c9d1be2837e0c3df4e86c6: Status 404 returned error can't find the container with id 8970e6d2934cec4a43d044e03239fb6b9b1b0e64a0c9d1be2837e0c3df4e86c6
Sep 19 08:21:38.821224 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.821202    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-hhscx"]
Sep 19 08:21:38.824027 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:21:38.823996    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60dc08bc_8f96_458d_a5a7_c09534ca4ef5.slice/crio-7bdf0cc72d534350c727a4e5681f394d6c39643299e047607d104b249294f689 WatchSource:0}: Error finding container 7bdf0cc72d534350c727a4e5681f394d6c39643299e047607d104b249294f689: Status 404 returned error can't find the container with id 7bdf0cc72d534350c727a4e5681f394d6c39643299e047607d104b249294f689
Sep 19 08:21:38.881618 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:38.881587    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64"]
Sep 19 08:21:38.893407 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:21:38.893373    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf453c92_08c7_44c0_9e10_52cf98b6b7f4.slice/crio-1f2c78222457bc2e39637a572487f6162910ef47cb30cef5f959f76e0497fb2e WatchSource:0}: Error finding container 1f2c78222457bc2e39637a572487f6162910ef47cb30cef5f959f76e0497fb2e: Status 404 returned error can't find the container with id 1f2c78222457bc2e39637a572487f6162910ef47cb30cef5f959f76e0497fb2e
Sep 19 08:21:39.072060 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:39.071964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:39.072202 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:39.072121    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:39.072202 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:39.072145    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcfbdb9c6-wc2zp: secret "image-registry-tls" not found
Sep 19 08:21:39.072326 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:39.072218    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls podName:b5bace44-66ba-4d16-8634-a518af242b6c nodeName:}" failed. No retries permitted until 2026-09-19 08:21:40.072194346 +0000 UTC m=+52.482717339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls") pod "image-registry-bcfbdb9c6-wc2zp" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c") : secret "image-registry-tls" not found
Sep 19 08:21:39.362696 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:39.362610    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gfznx" event={"ID":"cfe03657-f8db-4e34-90f3-e6762a4f8753","Type":"ContainerStarted","Data":"8970e6d2934cec4a43d044e03239fb6b9b1b0e64a0c9d1be2837e0c3df4e86c6"}
Sep 19 08:21:39.364244 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:39.364208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64" event={"ID":"af453c92-08c7-44c0-9e10-52cf98b6b7f4","Type":"ContainerStarted","Data":"1f2c78222457bc2e39637a572487f6162910ef47cb30cef5f959f76e0497fb2e"}
Sep 19 08:21:39.365752 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:39.365698    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-hhscx" event={"ID":"60dc08bc-8f96-458d-a5a7-c09534ca4ef5","Type":"ContainerStarted","Data":"7bdf0cc72d534350c727a4e5681f394d6c39643299e047607d104b249294f689"}
Sep 19 08:21:39.678402 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:39.678285    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:39.678402 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:39.678365    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls podName:6663b01e-cdec-4748-a1b3-7fd84903131d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:55.67833784 +0000 UTC m=+68.088860835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls") pod "dns-default-svmsl" (UID: "6663b01e-cdec-4748-a1b3-7fd84903131d") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:39.678705 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:39.678159    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:40.082236 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:40.082196    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:40.082425 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:40.082369    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:40.082425 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:40.082386    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcfbdb9c6-wc2zp: secret "image-registry-tls" not found
Sep 19 08:21:40.082530 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:40.082456    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls podName:b5bace44-66ba-4d16-8634-a518af242b6c nodeName:}" failed. No retries permitted until 2026-09-19 08:21:42.0824414 +0000 UTC m=+54.492964381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls") pod "image-registry-bcfbdb9c6-wc2zp" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c") : secret "image-registry-tls" not found
Sep 19 08:21:40.370974 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:40.370873    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gfznx" event={"ID":"cfe03657-f8db-4e34-90f3-e6762a4f8753","Type":"ContainerStarted","Data":"1d8a11d5efe1edbacab7a1eed1604216e5794e077a1ab4ea83d04e3fda4c24f2"}
Sep 19 08:21:40.386547 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:40.386482    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-gfznx" podStartSLOduration=1.010032939 podStartE2EDuration="2.386460077s" totalImagesPullingTime="1.376427138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:40.386156018 +0000 UTC m=+52.796679022" watchObservedRunningTime="2026-09-19 08:21:40.386460077 +0000 UTC m=+52.796983077"
Sep 19 08:21:40.500741 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:40.500704    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:40.600987 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:40.600943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:41.376077 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:41.376035    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64" event={"ID":"af453c92-08c7-44c0-9e10-52cf98b6b7f4","Type":"ContainerStarted","Data":"63fe1d343358a6a2ab4dca20f602604df95239193241c6758699d5a724727762"}
Sep 19 08:21:41.377737 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:41.377709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-hhscx" event={"ID":"60dc08bc-8f96-458d-a5a7-c09534ca4ef5","Type":"ContainerStarted","Data":"1c2e45a5c172bc8aa66e4ebda1e78e3bb3b12f09dcb02083661cc8ca49dafba4"}
Sep 19 08:21:41.393248 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:41.393208    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64" podStartSLOduration=1.094197964 podStartE2EDuration="3.393191087s" totalImagesPullingTime="2.298993123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:41.3920626 +0000 UTC m=+53.802585601" watchObservedRunningTime="2026-09-19 08:21:41.393191087 +0000 UTC m=+53.803714092"
Sep 19 08:21:41.411043 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:41.410377    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-hhscx" podStartSLOduration=1.043709503 podStartE2EDuration="3.410358625s" totalImagesPullingTime="2.366649122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:41.409624455 +0000 UTC m=+53.820147455" watchObservedRunningTime="2026-09-19 08:21:41.410358625 +0000 UTC m=+53.820881625"
Sep 19 08:21:42.098573 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:42.098534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:42.098755 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:42.098667    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:42.098755 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:42.098697    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcfbdb9c6-wc2zp: secret "image-registry-tls" not found
Sep 19 08:21:42.098831 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:42.098767    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls podName:b5bace44-66ba-4d16-8634-a518af242b6c nodeName:}" failed. No retries permitted until 2026-09-19 08:21:46.098746218 +0000 UTC m=+58.509269211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls") pod "image-registry-bcfbdb9c6-wc2zp" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c") : secret "image-registry-tls" not found
Sep 19 08:21:42.805999 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:42.805962    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:42.901595 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:42.901563    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:42.906352 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:42.906322    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t7vcg"]
Sep 19 08:21:42.908599 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:42.908581    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:42.911131 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:42.911101    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k69r7\""
Sep 19 08:21:42.911276 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:42.911207    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 08:21:42.912139 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:42.912126    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 08:21:42.924108 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:42.924085    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t7vcg"]
Sep 19 08:21:43.005932 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.005897    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c64a1e78-7b83-4797-baca-6f3ccbea5a53-crio-socket\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.006113 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.005950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.006113 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.005985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c64a1e78-7b83-4797-baca-6f3ccbea5a53-data-volume\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.006113 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.006036    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2v2r\" (UniqueName: \"kubernetes.io/projected/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-api-access-v2v2r\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.006218 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.006123    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.013945 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.013909    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:43.107107 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.107010    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2v2r\" (UniqueName: \"kubernetes.io/projected/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-api-access-v2v2r\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.107107 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.107079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.107107 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.107099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c64a1e78-7b83-4797-baca-6f3ccbea5a53-crio-socket\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.107469 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.107138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.107469 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.107160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c64a1e78-7b83-4797-baca-6f3ccbea5a53-data-volume\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.107469 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.107238    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c64a1e78-7b83-4797-baca-6f3ccbea5a53-crio-socket\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.107469 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:43.107293    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:43.107469 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:43.107362    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls podName:c64a1e78-7b83-4797-baca-6f3ccbea5a53 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:43.607345971 +0000 UTC m=+56.017868949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7vcg" (UID: "c64a1e78-7b83-4797-baca-6f3ccbea5a53") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:43.107469 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:43.107293    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:43.107469 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:43.107438    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm podName:c64a1e78-7b83-4797-baca-6f3ccbea5a53 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:43.607425849 +0000 UTC m=+56.017948869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t7vcg" (UID: "c64a1e78-7b83-4797-baca-6f3ccbea5a53") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:43.107469 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.107472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c64a1e78-7b83-4797-baca-6f3ccbea5a53-data-volume\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.117525 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.117504    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2v2r\" (UniqueName: \"kubernetes.io/projected/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-api-access-v2v2r\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.211353 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.211318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:43.264651 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.264619    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:43.610977 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.610934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.611189 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.610992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:43.611189 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:43.611063    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:43.611189 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:43.611102    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:43.611189 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:43.611126    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm podName:c64a1e78-7b83-4797-baca-6f3ccbea5a53 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:44.61111163 +0000 UTC m=+57.021634608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t7vcg" (UID: "c64a1e78-7b83-4797-baca-6f3ccbea5a53") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:43.611189 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:43.611163    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls podName:c64a1e78-7b83-4797-baca-6f3ccbea5a53 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:44.61114645 +0000 UTC m=+57.021669430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7vcg" (UID: "c64a1e78-7b83-4797-baca-6f3ccbea5a53") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:43.673699 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.673666    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:43.772586 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.772553    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:43.973207 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:43.973129    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:44.064487 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:44.064454    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:44.619775 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:44.619742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:44.619931 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:44.619793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:44.619931 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:44.619895    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:44.619931 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:44.619901    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:44.620050 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:44.619947    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm podName:c64a1e78-7b83-4797-baca-6f3ccbea5a53 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:46.619934591 +0000 UTC m=+59.030457570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t7vcg" (UID: "c64a1e78-7b83-4797-baca-6f3ccbea5a53") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:44.620050 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:44.619965    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls podName:c64a1e78-7b83-4797-baca-6f3ccbea5a53 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:46.619955747 +0000 UTC m=+59.030478724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7vcg" (UID: "c64a1e78-7b83-4797-baca-6f3ccbea5a53") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:45.032893 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.032872    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xn4mf_af211ba6-f358-4565-a7dd-37eb2337ea07/dns-node-resolver/0.log"
Sep 19 08:21:45.255765 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.255733    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:45.358336 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.358229    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:45.535303 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.535267    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-gqnnr"]
Sep 19 08:21:45.539196 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.539179    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr"
Sep 19 08:21:45.542280 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.542237    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 19 08:21:45.542406 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.542323    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 19 08:21:45.543169 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.543152    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 19 08:21:45.543361 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.543340    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 19 08:21:45.543361 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.543351    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-287rl\""
Sep 19 08:21:45.548564 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.548546    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-gqnnr"]
Sep 19 08:21:45.579075 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.579047    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:45.628187 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.628119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/787a7847-273c-4655-b634-8bdf123af303-signing-key\") pod \"service-ca-55f6fc7b86-gqnnr\" (UID: \"787a7847-273c-4655-b634-8bdf123af303\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr"
Sep 19 08:21:45.628332 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.628203    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rg8j\" (UniqueName: \"kubernetes.io/projected/787a7847-273c-4655-b634-8bdf123af303-kube-api-access-8rg8j\") pod \"service-ca-55f6fc7b86-gqnnr\" (UID: \"787a7847-273c-4655-b634-8bdf123af303\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr"
Sep 19 08:21:45.628332 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.628286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/787a7847-273c-4655-b634-8bdf123af303-signing-cabundle\") pod \"service-ca-55f6fc7b86-gqnnr\" (UID: \"787a7847-273c-4655-b634-8bdf123af303\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr"
Sep 19 08:21:45.642577 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.642551    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:45.729174 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.729134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rg8j\" (UniqueName: \"kubernetes.io/projected/787a7847-273c-4655-b634-8bdf123af303-kube-api-access-8rg8j\") pod \"service-ca-55f6fc7b86-gqnnr\" (UID: \"787a7847-273c-4655-b634-8bdf123af303\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr"
Sep 19 08:21:45.729387 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.729242    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/787a7847-273c-4655-b634-8bdf123af303-signing-cabundle\") pod \"service-ca-55f6fc7b86-gqnnr\" (UID: \"787a7847-273c-4655-b634-8bdf123af303\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr"
Sep 19 08:21:45.729387 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.729324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/787a7847-273c-4655-b634-8bdf123af303-signing-key\") pod \"service-ca-55f6fc7b86-gqnnr\" (UID: \"787a7847-273c-4655-b634-8bdf123af303\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr"
Sep 19 08:21:45.729924 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.729899    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/787a7847-273c-4655-b634-8bdf123af303-signing-cabundle\") pod \"service-ca-55f6fc7b86-gqnnr\" (UID: \"787a7847-273c-4655-b634-8bdf123af303\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr"
Sep 19 08:21:45.731732 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.731703    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/787a7847-273c-4655-b634-8bdf123af303-signing-key\") pod \"service-ca-55f6fc7b86-gqnnr\" (UID: \"787a7847-273c-4655-b634-8bdf123af303\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr"
Sep 19 08:21:45.738082 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.738057    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rg8j\" (UniqueName: \"kubernetes.io/projected/787a7847-273c-4655-b634-8bdf123af303-kube-api-access-8rg8j\") pod \"service-ca-55f6fc7b86-gqnnr\" (UID: \"787a7847-273c-4655-b634-8bdf123af303\") " pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr"
Sep 19 08:21:45.832115 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.832084    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lcmrp_2d618e83-8b27-4401-94d2-1feaa04c634f/node-ca/0.log"
Sep 19 08:21:45.848175 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:45.848149    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr"
Sep 19 08:21:46.132763 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:46.132717    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:46.133136 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:46.132873    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 19 08:21:46.133136 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:46.132893    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcfbdb9c6-wc2zp: secret "image-registry-tls" not found
Sep 19 08:21:46.133136 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:46.132951    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls podName:b5bace44-66ba-4d16-8634-a518af242b6c nodeName:}" failed. No retries permitted until 2026-09-19 08:21:54.132934047 +0000 UTC m=+66.543457025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls") pod "image-registry-bcfbdb9c6-wc2zp" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c") : secret "image-registry-tls" not found
Sep 19 08:21:46.146576 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:46.144625    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-gqnnr"]
Sep 19 08:21:46.149558 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:21:46.149534    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787a7847_273c_4655_b634_8bdf123af303.slice/crio-e3f0184b82e1eedafd0c72fa1fbf407e448cdc537cd67151b42825a1eba0a595 WatchSource:0}: Error finding container e3f0184b82e1eedafd0c72fa1fbf407e448cdc537cd67151b42825a1eba0a595: Status 404 returned error can't find the container with id e3f0184b82e1eedafd0c72fa1fbf407e448cdc537cd67151b42825a1eba0a595
Sep 19 08:21:46.392250 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:46.392209    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr" event={"ID":"787a7847-273c-4655-b634-8bdf123af303","Type":"ContainerStarted","Data":"e3f0184b82e1eedafd0c72fa1fbf407e448cdc537cd67151b42825a1eba0a595"}
Sep 19 08:21:46.433116 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:46.433083    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-45sdz_8098f93f-f309-4e66-a1a5-fe28c2b8a5d8/serve-healthcheck-canary/0.log"
Sep 19 08:21:46.614749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:46.614706    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:46.638106 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:46.638069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:46.638335 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:46.638145    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:46.638335 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:46.638250    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:46.638335 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:46.638323    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm podName:c64a1e78-7b83-4797-baca-6f3ccbea5a53 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:50.638305729 +0000 UTC m=+63.048828727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t7vcg" (UID: "c64a1e78-7b83-4797-baca-6f3ccbea5a53") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:46.638684 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:46.638657    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:46.638803 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:46.638731    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls podName:c64a1e78-7b83-4797-baca-6f3ccbea5a53 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:50.638712119 +0000 UTC m=+63.049235098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7vcg" (UID: "c64a1e78-7b83-4797-baca-6f3ccbea5a53") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:48.399776 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:48.399739    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr" event={"ID":"787a7847-273c-4655-b634-8bdf123af303","Type":"ContainerStarted","Data":"8a42edfbd1753e0d720c1ffc12b52f071cc939256cfc720ece137257b482453a"}
Sep 19 08:21:48.417040 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:48.416985    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-gqnnr" podStartSLOduration=1.6706076730000001 podStartE2EDuration="3.416969191s" totalImagesPullingTime="1.746361518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:48.416151614 +0000 UTC m=+60.826674615" watchObservedRunningTime="2026-09-19 08:21:48.416969191 +0000 UTC m=+60.827492188"
Sep 19 08:21:48.979645 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:48.979605    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:49.306028 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:49.305996    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnft5"
Sep 19 08:21:49.307856 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:49.307818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:49.442663 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:49.442632    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:49.546778 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:49.546743    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:49.669103 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:49.669015    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:49.763942 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:49.763905    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:50.151030 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:50.150995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:50.296336 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:50.296296    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:50.673867 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:50.673822    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:50.674317 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:50.673902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:50.674317 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:50.673989    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:50.674317 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:50.674016    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:50.674317 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:50.674071    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls podName:c64a1e78-7b83-4797-baca-6f3ccbea5a53 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:58.674049064 +0000 UTC m=+71.084572048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7vcg" (UID: "c64a1e78-7b83-4797-baca-6f3ccbea5a53") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:50.674317 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:50.674092    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm podName:c64a1e78-7b83-4797-baca-6f3ccbea5a53 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:58.674082176 +0000 UTC m=+71.084605160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t7vcg" (UID: "c64a1e78-7b83-4797-baca-6f3ccbea5a53") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:50.693876 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:50.693846    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:51.287935 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:51.287898    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:51.687653 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:51.687570    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:51.893640 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:51.893611    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:52.288579 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:52.288542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:52.661465 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:52.661388    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:52.815435 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:52.815399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:52.921957 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:52.921869    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:53.283306 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:53.283270    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:53.493735 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:53.493701    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:53.570837 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:53.570756    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:53.898068 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:53.897989    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:54.206926 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.206831    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:54.209280 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.209235    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls\") pod \"image-registry-bcfbdb9c6-wc2zp\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") " pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:54.289122 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.289084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:54.499227 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.499153    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v4wbb\""
Sep 19 08:21:54.507588 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.507567    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:54.711271 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.711216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:21:54.711453 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.711293    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:54.714160 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.714141    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:54.714234 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.714180    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:54.722489 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:54.722456    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 19 08:21:54.722653 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:54.722530    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs podName:9178e60e-1f54-461c-8a40-1b8c3300b64c nodeName:}" failed. No retries permitted until 2026-09-19 08:22:58.722511738 +0000 UTC m=+131.133034729 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs") pod "network-metrics-daemon-7qn5v" (UID: "9178e60e-1f54-461c-8a40-1b8c3300b64c") : secret "metrics-daemon-secret" not found
Sep 19 08:21:54.724753 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.724732    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/078bb266-8297-48c9-a45b-2d47fb7cbd45-original-pull-secret\") pod \"global-pull-secret-syncer-xvkt5\" (UID: \"078bb266-8297-48c9-a45b-2d47fb7cbd45\") " pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:54.779446 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.779371    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"]
Sep 19 08:21:54.783650 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:21:54.783624    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bace44_66ba_4d16_8634_a518af242b6c.slice/crio-b97c9978200f8af146fca9fbe3de54c1f3634fb20eaf148e7909d41b3770eae1 WatchSource:0}: Error finding container b97c9978200f8af146fca9fbe3de54c1f3634fb20eaf148e7909d41b3770eae1: Status 404 returned error can't find the container with id b97c9978200f8af146fca9fbe3de54c1f3634fb20eaf148e7909d41b3770eae1
Sep 19 08:21:54.812717 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.812691    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pdb9\" (UniqueName: \"kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9\") pod \"network-check-target-gcm28\" (UID: \"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1\") " pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:54.815367 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.815342    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 08:21:54.825793 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.825770    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 08:21:54.836692 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.836668    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pdb9\" (UniqueName: \"kubernetes.io/projected/4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1-kube-api-access-2pdb9\") pod \"network-check-target-gcm28\" (UID: \"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1\") " pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:54.944561 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.944509    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xvkt5"
Sep 19 08:21:54.960448 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.960418    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47kpw\""
Sep 19 08:21:54.968674 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:54.968645    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:21:55.094277 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.094227    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:55.220817 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.220785    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xvkt5"]
Sep 19 08:21:55.224354 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:21:55.224320    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078bb266_8297_48c9_a45b_2d47fb7cbd45.slice/crio-88c3e7c9a6d8259046fa8a72a5165b5673b67d7b46c3ff4794eeb478cc662895 WatchSource:0}: Error finding container 88c3e7c9a6d8259046fa8a72a5165b5673b67d7b46c3ff4794eeb478cc662895: Status 404 returned error can't find the container with id 88c3e7c9a6d8259046fa8a72a5165b5673b67d7b46c3ff4794eeb478cc662895
Sep 19 08:21:55.244775 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.244750    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gcm28"]
Sep 19 08:21:55.247012 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:21:55.246986    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca7dd96_1b7a_462b_9d32_8c9f89fea1e1.slice/crio-032db61e50a6d53e09c02d167984b9d7f8de2a6cd391a2d3ca8ba163229b8f0b WatchSource:0}: Error finding container 032db61e50a6d53e09c02d167984b9d7f8de2a6cd391a2d3ca8ba163229b8f0b: Status 404 returned error can't find the container with id 032db61e50a6d53e09c02d167984b9d7f8de2a6cd391a2d3ca8ba163229b8f0b
Sep 19 08:21:55.275414 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.275379    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:55.428128 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.428048    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xvkt5" event={"ID":"078bb266-8297-48c9-a45b-2d47fb7cbd45","Type":"ContainerStarted","Data":"88c3e7c9a6d8259046fa8a72a5165b5673b67d7b46c3ff4794eeb478cc662895"}
Sep 19 08:21:55.429099 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.429070    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gcm28" event={"ID":"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1","Type":"ContainerStarted","Data":"032db61e50a6d53e09c02d167984b9d7f8de2a6cd391a2d3ca8ba163229b8f0b"}
Sep 19 08:21:55.430478 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.430459    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp" event={"ID":"b5bace44-66ba-4d16-8634-a518af242b6c","Type":"ContainerStarted","Data":"64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7"}
Sep 19 08:21:55.430576 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.430494    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp" event={"ID":"b5bace44-66ba-4d16-8634-a518af242b6c","Type":"ContainerStarted","Data":"b97c9978200f8af146fca9fbe3de54c1f3634fb20eaf148e7909d41b3770eae1"}
Sep 19 08:21:55.430633 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.430604    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:21:55.453070 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.453015    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp" podStartSLOduration=17.452998205 podStartE2EDuration="17.452998205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:55.452204694 +0000 UTC m=+67.862727699" watchObservedRunningTime="2026-09-19 08:21:55.452998205 +0000 UTC m=+67.863521205"
Sep 19 08:21:55.488831 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.488784    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:55.688407 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.688323    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:55.719513 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.719477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:55.722414 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.722389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6663b01e-cdec-4748-a1b3-7fd84903131d-metrics-tls\") pod \"dns-default-svmsl\" (UID: \"6663b01e-cdec-4748-a1b3-7fd84903131d\") " pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:55.895521 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.895478    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:55.977831 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.977744    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g2bvg\""
Sep 19 08:21:55.986030 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:55.985771    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-svmsl"
Sep 19 08:21:56.002305 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:56.002130    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:56.287121 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:56.287079    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:56.302409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:56.302380    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-svmsl"]
Sep 19 08:21:56.305145 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:21:56.305113    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6663b01e_cdec_4748_a1b3_7fd84903131d.slice/crio-096cf6f6d999e138e368c967676bf469b736cec553a40331045ec10cc9bd9fdc WatchSource:0}: Error finding container 096cf6f6d999e138e368c967676bf469b736cec553a40331045ec10cc9bd9fdc: Status 404 returned error can't find the container with id 096cf6f6d999e138e368c967676bf469b736cec553a40331045ec10cc9bd9fdc
Sep 19 08:21:56.437793 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:56.437704    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-svmsl" event={"ID":"6663b01e-cdec-4748-a1b3-7fd84903131d","Type":"ContainerStarted","Data":"096cf6f6d999e138e368c967676bf469b736cec553a40331045ec10cc9bd9fdc"}
Sep 19 08:21:56.689910 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:56.689748    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:57.316969 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:57.316928    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:57.615734 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:57.615624    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:57.716568 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:57.716471    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:58.095963 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:58.095896    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:58.496426 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:58.496325    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:58.688565 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:58.688534    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:58.745932 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:58.745890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:58.746142 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:58.745974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:58.746142 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:58.746079    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:58.746284 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:21:58.746162    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls podName:c64a1e78-7b83-4797-baca-6f3ccbea5a53 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:14.746140486 +0000 UTC m=+87.156663477 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t7vcg" (UID: "c64a1e78-7b83-4797-baca-6f3ccbea5a53") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:58.746654 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:58.746592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c64a1e78-7b83-4797-baca-6f3ccbea5a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:21:59.091750 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:59.091714    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:59.491123 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:59.491031    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:21:59.687271 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:21:59.687222    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:00.364427 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:00.364390    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:00.457089 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:00.457057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gcm28" event={"ID":"4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1","Type":"ContainerStarted","Data":"49fff8462138a694dc13c886c54efff9ada7738d02dc4327766988d7ac55e857"}
Sep 19 08:22:00.457353 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:00.457301    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:22:00.473153 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:00.473112    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gcm28" podStartSLOduration=65.346977455 podStartE2EDuration="1m10.473097077s" totalImagesPullingTime="5.126119622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:00.471933016 +0000 UTC m=+72.882456016" watchObservedRunningTime="2026-09-19 08:22:00.473097077 +0000 UTC m=+72.883620119"
Sep 19 08:22:00.473982 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:00.473964    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:00.675383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:00.675288    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:01.044402 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:01.044369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:01.104873 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:01.104840    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:01.282558 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:01.282524    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:01.386206 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:01.386133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:01.462225 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:01.462189    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xvkt5" event={"ID":"078bb266-8297-48c9-a45b-2d47fb7cbd45","Type":"ContainerStarted","Data":"92dbc636877497993f3d112a12dfc3147e1e93ebd79fc876f759fdd1a2e43738"}
Sep 19 08:22:01.464237 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:01.464214    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-svmsl" event={"ID":"6663b01e-cdec-4748-a1b3-7fd84903131d","Type":"ContainerStarted","Data":"75960dfbb9695f1ce7458beeb5c42b54a692ff76a4cbbf80632caaa75408a2e9"}
Sep 19 08:22:01.464377 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:01.464272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-svmsl" event={"ID":"6663b01e-cdec-4748-a1b3-7fd84903131d","Type":"ContainerStarted","Data":"59c724e79d6a80114d0cdbd87c9170e6db5c44f016a9664ba0a86707c66ea784"}
Sep 19 08:22:01.512078 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:01.512023    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xvkt5" podStartSLOduration=66.349651856 podStartE2EDuration="1m11.512002977s" totalImagesPullingTime="5.162351121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:01.48223467 +0000 UTC m=+73.892757670" watchObservedRunningTime="2026-09-19 08:22:01.512002977 +0000 UTC m=+73.922525976"
Sep 19 08:22:01.512817 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:01.512781    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-svmsl" podStartSLOduration=34.443201317 podStartE2EDuration="38.512766103s" totalImagesPullingTime="4.069564786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:01.510892052 +0000 UTC m=+73.921415053" watchObservedRunningTime="2026-09-19 08:22:01.512766103 +0000 UTC m=+73.923289173"
Sep 19 08:22:02.294148 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:02.294111    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:02.467787 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:02.467755    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-svmsl"
Sep 19 08:22:02.509600 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:02.509567    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:02.785702 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:02.785673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:03.086245 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:03.086162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:04.751355 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:04.751322    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:04.821287 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:04.821235    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:05.685728 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.685677    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"]
Sep 19 08:22:05.688913 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.688896    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:05.692075 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.692052    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 19 08:22:05.693199 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.693170    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 19 08:22:05.693357 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.693215    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 19 08:22:05.695217 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.695195    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 19 08:22:05.709441 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.709414    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"]
Sep 19 08:22:05.788155 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.788113    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-bqfng"]
Sep 19 08:22:05.792389 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.792286    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-bqfng"
Sep 19 08:22:05.796356 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.796332    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 19 08:22:05.796493 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.796455    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"]
Sep 19 08:22:05.796588 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.796568    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 19 08:22:05.796651 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.796625    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2kkkz\""
Sep 19 08:22:05.796738 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.796722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrg99\" (UniqueName: \"kubernetes.io/projected/0a3ef174-02df-483b-86ef-a3a764df6f5d-kube-api-access-vrg99\") pod \"klusterlet-addon-workmgr-54d6cf5c78-dvrqt\" (UID: \"0a3ef174-02df-483b-86ef-a3a764df6f5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:05.796811 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.796770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3ef174-02df-483b-86ef-a3a764df6f5d-tmp\") pod \"klusterlet-addon-workmgr-54d6cf5c78-dvrqt\" (UID: \"0a3ef174-02df-483b-86ef-a3a764df6f5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:05.796878 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.796854    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0a3ef174-02df-483b-86ef-a3a764df6f5d-klusterlet-config\") pod \"klusterlet-addon-workmgr-54d6cf5c78-dvrqt\" (UID: \"0a3ef174-02df-483b-86ef-a3a764df6f5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:05.825525 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.825493    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-bqfng"]
Sep 19 08:22:05.877943 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.877907    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c6758476b-lz964"]
Sep 19 08:22:05.880991 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.880968    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:05.897936 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.897903    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrg99\" (UniqueName: \"kubernetes.io/projected/0a3ef174-02df-483b-86ef-a3a764df6f5d-kube-api-access-vrg99\") pod \"klusterlet-addon-workmgr-54d6cf5c78-dvrqt\" (UID: \"0a3ef174-02df-483b-86ef-a3a764df6f5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:05.898079 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.897954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ffj5\" (UniqueName: \"kubernetes.io/projected/d335ad1a-3686-4a3f-9ef8-88f7989199e6-kube-api-access-7ffj5\") pod \"downloads-6594f4c75-bqfng\" (UID: \"d335ad1a-3686-4a3f-9ef8-88f7989199e6\") " pod="openshift-console/downloads-6594f4c75-bqfng"
Sep 19 08:22:05.898079 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.898036    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3ef174-02df-483b-86ef-a3a764df6f5d-tmp\") pod \"klusterlet-addon-workmgr-54d6cf5c78-dvrqt\" (UID: \"0a3ef174-02df-483b-86ef-a3a764df6f5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:05.898197 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.898096    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0a3ef174-02df-483b-86ef-a3a764df6f5d-klusterlet-config\") pod \"klusterlet-addon-workmgr-54d6cf5c78-dvrqt\" (UID: \"0a3ef174-02df-483b-86ef-a3a764df6f5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:05.898482 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.898459    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3ef174-02df-483b-86ef-a3a764df6f5d-tmp\") pod \"klusterlet-addon-workmgr-54d6cf5c78-dvrqt\" (UID: \"0a3ef174-02df-483b-86ef-a3a764df6f5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:05.899351 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.899331    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c6758476b-lz964"]
Sep 19 08:22:05.900578 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.900557    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:05.900995 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.900973    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0a3ef174-02df-483b-86ef-a3a764df6f5d-klusterlet-config\") pod \"klusterlet-addon-workmgr-54d6cf5c78-dvrqt\" (UID: \"0a3ef174-02df-483b-86ef-a3a764df6f5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:05.915062 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.915039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrg99\" (UniqueName: \"kubernetes.io/projected/0a3ef174-02df-483b-86ef-a3a764df6f5d-kube-api-access-vrg99\") pod \"klusterlet-addon-workmgr-54d6cf5c78-dvrqt\" (UID: \"0a3ef174-02df-483b-86ef-a3a764df6f5d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:05.998907 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.998798    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:05.999077 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.998972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a03882a4-c7f1-41d9-932b-54efda96e8ab-registry-certificates\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:05.999077 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.999012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a03882a4-c7f1-41d9-932b-54efda96e8ab-trusted-ca\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:05.999077 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.999057    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a03882a4-c7f1-41d9-932b-54efda96e8ab-registry-tls\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:05.999192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.999087    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a03882a4-c7f1-41d9-932b-54efda96e8ab-bound-sa-token\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:05.999192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.999113    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ffj5\" (UniqueName: \"kubernetes.io/projected/d335ad1a-3686-4a3f-9ef8-88f7989199e6-kube-api-access-7ffj5\") pod \"downloads-6594f4c75-bqfng\" (UID: \"d335ad1a-3686-4a3f-9ef8-88f7989199e6\") " pod="openshift-console/downloads-6594f4c75-bqfng"
Sep 19 08:22:05.999192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.999130    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a03882a4-c7f1-41d9-932b-54efda96e8ab-installation-pull-secrets\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:05.999192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.999182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a03882a4-c7f1-41d9-932b-54efda96e8ab-image-registry-private-configuration\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:05.999361 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.999203    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qhn\" (UniqueName: \"kubernetes.io/projected/a03882a4-c7f1-41d9-932b-54efda96e8ab-kube-api-access-49qhn\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:05.999361 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:05.999228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a03882a4-c7f1-41d9-932b-54efda96e8ab-ca-trust-extracted\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.006674 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.006653    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:06.010824 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.010803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ffj5\" (UniqueName: \"kubernetes.io/projected/d335ad1a-3686-4a3f-9ef8-88f7989199e6-kube-api-access-7ffj5\") pod \"downloads-6594f4c75-bqfng\" (UID: \"d335ad1a-3686-4a3f-9ef8-88f7989199e6\") " pod="openshift-console/downloads-6594f4c75-bqfng"
Sep 19 08:22:06.099874 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.099837    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a03882a4-c7f1-41d9-932b-54efda96e8ab-image-registry-private-configuration\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.099874 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.099878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qhn\" (UniqueName: \"kubernetes.io/projected/a03882a4-c7f1-41d9-932b-54efda96e8ab-kube-api-access-49qhn\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.100112 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.099908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a03882a4-c7f1-41d9-932b-54efda96e8ab-ca-trust-extracted\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.100112 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.099977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a03882a4-c7f1-41d9-932b-54efda96e8ab-registry-certificates\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.100112 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.100005    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a03882a4-c7f1-41d9-932b-54efda96e8ab-trusted-ca\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.100112 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.100028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a03882a4-c7f1-41d9-932b-54efda96e8ab-registry-tls\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.100112 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.100051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a03882a4-c7f1-41d9-932b-54efda96e8ab-bound-sa-token\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.100112 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.100079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a03882a4-c7f1-41d9-932b-54efda96e8ab-installation-pull-secrets\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.100581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.100562    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a03882a4-c7f1-41d9-932b-54efda96e8ab-ca-trust-extracted\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.100908 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.100888    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a03882a4-c7f1-41d9-932b-54efda96e8ab-registry-certificates\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.101172 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.101151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a03882a4-c7f1-41d9-932b-54efda96e8ab-trusted-ca\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.102473 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.102450    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-bqfng"
Sep 19 08:22:06.102664 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.102643    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a03882a4-c7f1-41d9-932b-54efda96e8ab-installation-pull-secrets\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.102887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.102869    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a03882a4-c7f1-41d9-932b-54efda96e8ab-image-registry-private-configuration\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.102951 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.102872    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a03882a4-c7f1-41d9-932b-54efda96e8ab-registry-tls\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.109633 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.109613    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qhn\" (UniqueName: \"kubernetes.io/projected/a03882a4-c7f1-41d9-932b-54efda96e8ab-kube-api-access-49qhn\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.112905 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.112889    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a03882a4-c7f1-41d9-932b-54efda96e8ab-bound-sa-token\") pod \"image-registry-7c6758476b-lz964\" (UID: \"a03882a4-c7f1-41d9-932b-54efda96e8ab\") " pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.142263 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.142218    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:06.189650 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.189621    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:06.251404 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.251334    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:06.317665 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:22:06.317639    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3ef174_02df_483b_86ef_a3a764df6f5d.slice/crio-3c60a1468b278cf844110be812005213b5d3d90de19f0af21e7d9a0033b34b85 WatchSource:0}: Error finding container 3c60a1468b278cf844110be812005213b5d3d90de19f0af21e7d9a0033b34b85: Status 404 returned error can't find the container with id 3c60a1468b278cf844110be812005213b5d3d90de19f0af21e7d9a0033b34b85
Sep 19 08:22:06.324507 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.324485    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"]
Sep 19 08:22:06.393282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.393216    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-bqfng"]
Sep 19 08:22:06.397803 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:22:06.397771    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd335ad1a_3686_4a3f_9ef8_88f7989199e6.slice/crio-e331db39b52430eb0f9f1c009ac70a6d3172d3315344545c2a021c71dfdbb05d WatchSource:0}: Error finding container e331db39b52430eb0f9f1c009ac70a6d3172d3315344545c2a021c71dfdbb05d: Status 404 returned error can't find the container with id e331db39b52430eb0f9f1c009ac70a6d3172d3315344545c2a021c71dfdbb05d
Sep 19 08:22:06.471782 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.471644    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c6758476b-lz964"]
Sep 19 08:22:06.474329 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:22:06.474302    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03882a4_c7f1_41d9_932b_54efda96e8ab.slice/crio-00b1ef0c61e6611e486d1e742ef2983474afa68bdd79e1bb7afc4af5e186cfc7 WatchSource:0}: Error finding container 00b1ef0c61e6611e486d1e742ef2983474afa68bdd79e1bb7afc4af5e186cfc7: Status 404 returned error can't find the container with id 00b1ef0c61e6611e486d1e742ef2983474afa68bdd79e1bb7afc4af5e186cfc7
Sep 19 08:22:06.480402 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.480374    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt" event={"ID":"0a3ef174-02df-483b-86ef-a3a764df6f5d","Type":"ContainerStarted","Data":"3c60a1468b278cf844110be812005213b5d3d90de19f0af21e7d9a0033b34b85"}
Sep 19 08:22:06.481656 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.481521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c6758476b-lz964" event={"ID":"a03882a4-c7f1-41d9-932b-54efda96e8ab","Type":"ContainerStarted","Data":"00b1ef0c61e6611e486d1e742ef2983474afa68bdd79e1bb7afc4af5e186cfc7"}
Sep 19 08:22:06.485071 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:06.485046    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-bqfng" event={"ID":"d335ad1a-3686-4a3f-9ef8-88f7989199e6","Type":"ContainerStarted","Data":"e331db39b52430eb0f9f1c009ac70a6d3172d3315344545c2a021c71dfdbb05d"}
Sep 19 08:22:07.493279 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:07.492911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c6758476b-lz964" event={"ID":"a03882a4-c7f1-41d9-932b-54efda96e8ab","Type":"ContainerStarted","Data":"1ef72d0acae67f1de39988470d0f10f0a7fe6c2ba062e18dcdac3d37960e1d86"}
Sep 19 08:22:07.493279 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:07.493025    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:07.522006 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:07.521712    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c6758476b-lz964" podStartSLOduration=2.52168789 podStartE2EDuration="2.52168789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:07.52068385 +0000 UTC m=+79.931206851" watchObservedRunningTime="2026-09-19 08:22:07.52168789 +0000 UTC m=+79.932210891"
Sep 19 08:22:11.479906 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:11.479872    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:11.512421 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:11.512378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt" event={"ID":"0a3ef174-02df-483b-86ef-a3a764df6f5d","Type":"ContainerStarted","Data":"66a44db4098f458fca7eb9acce09a8f83790c9d30f960ff215836d6d18933996"}
Sep 19 08:22:11.512671 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:11.512649    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:11.514373 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:11.514352    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt"
Sep 19 08:22:11.528579 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:11.528519    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d6cf5c78-dvrqt" podStartSLOduration=2.195473474 podStartE2EDuration="6.528502337s" totalImagesPullingTime="4.333028863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:11.528165781 +0000 UTC m=+83.938688782" watchObservedRunningTime="2026-09-19 08:22:11.528502337 +0000 UTC m=+83.939025337"
Sep 19 08:22:11.545269 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:11.545221    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:12.069738 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:12.069703    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:12.171566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:12.171531    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:12.473086 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:12.473003    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-svmsl"
Sep 19 08:22:12.637402 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:12.637364    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:12.737460 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:12.737348    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:12.859413 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:12.859378    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:13.090162 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:13.090125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:13.480981 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:13.480889    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:13.539623 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:13.539589    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:14.778044 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:14.778008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:22:14.780767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:14.780743    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c64a1e78-7b83-4797-baca-6f3ccbea5a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t7vcg\" (UID: \"c64a1e78-7b83-4797-baca-6f3ccbea5a53\") " pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:22:15.019806 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:15.019772    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k69r7\""
Sep 19 08:22:15.027816 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:15.027790    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t7vcg"
Sep 19 08:22:15.307647 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:15.307575    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t7vcg"]
Sep 19 08:22:15.310899 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:22:15.310862    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64a1e78_7b83_4797_baca_6f3ccbea5a53.slice/crio-cdc79ad13ee00a7ca2e70808b58fd847e746f191db9e00d6bb0a9d62e19e02af WatchSource:0}: Error finding container cdc79ad13ee00a7ca2e70808b58fd847e746f191db9e00d6bb0a9d62e19e02af: Status 404 returned error can't find the container with id cdc79ad13ee00a7ca2e70808b58fd847e746f191db9e00d6bb0a9d62e19e02af
Sep 19 08:22:15.484581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:15.484464    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:15.529912 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:15.529867    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7vcg" event={"ID":"c64a1e78-7b83-4797-baca-6f3ccbea5a53","Type":"ContainerStarted","Data":"9869f85299eb5792207d30fadf4be0317c0185aedf9a2989ef2dbd7e2b17162c"}
Sep 19 08:22:15.530089 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:15.529943    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7vcg" event={"ID":"c64a1e78-7b83-4797-baca-6f3ccbea5a53","Type":"ContainerStarted","Data":"cdc79ad13ee00a7ca2e70808b58fd847e746f191db9e00d6bb0a9d62e19e02af"}
Sep 19 08:22:15.585860 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:15.585783    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:15.803899 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:15.803679    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:22:16.835726 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:16.835686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:16.907408 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:16.907366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:16.987107 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:16.987050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:18.309878 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:18.309818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:18.410349 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:18.410309    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:18.471335 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:18.471246    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:19.012499 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:19.012463    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:19.112856 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:19.112819    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:19.909967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:19.909934    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:20.009757 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:20.009726    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:20.677453 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:20.677417    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:20.739154 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:20.739114    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:21.802970 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:21.802936    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:23.559558 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:23.559505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-bqfng" event={"ID":"d335ad1a-3686-4a3f-9ef8-88f7989199e6","Type":"ContainerStarted","Data":"0b9b68583e6b33156f29363e4eb1a871d074f219a25e47b33164546d7a07f445"}
Sep 19 08:22:23.560064 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:23.559608    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-bqfng"
Sep 19 08:22:23.561390 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:23.561359    2559 patch_prober.go:28] interesting pod/downloads-6594f4c75-bqfng container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" start-of-body=
Sep 19 08:22:23.561514 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:23.561417    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6594f4c75-bqfng" podUID="d335ad1a-3686-4a3f-9ef8-88f7989199e6" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused"
Sep 19 08:22:23.561697 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:23.561642    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7vcg" event={"ID":"c64a1e78-7b83-4797-baca-6f3ccbea5a53","Type":"ContainerStarted","Data":"fa098f2cc421d0c0cf60f5e9de084b424482050b76ef88e99288c9217ecf5a77"}
Sep 19 08:22:23.579811 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:23.579767    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-bqfng" podStartSLOduration=1.7345357 podStartE2EDuration="18.579751345s" totalImagesPullingTime="16.845215645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:23.579293249 +0000 UTC m=+95.989816249" watchObservedRunningTime="2026-09-19 08:22:23.579751345 +0000 UTC m=+95.990274345"
Sep 19 08:22:23.933819 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:23.933703    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:24.038672 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.038580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:24.199242 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.199152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:24.328770 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.328732    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:24.356560 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.355119    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"]
Sep 19 08:22:24.377910 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.377878    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"]
Sep 19 08:22:24.378089 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.378019    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.378662 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.378026    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-dpq85"]
Sep 19 08:22:24.382639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.381555    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 19 08:22:24.382639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.381760    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 19 08:22:24.382639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.381928    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-g8k7j\""
Sep 19 08:22:24.382639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.382430    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 19 08:22:24.383456 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.383102    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 19 08:22:24.383456 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.383338    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 19 08:22:24.393758 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.393436    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.399836 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.398858    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7x5qt\""
Sep 19 08:22:24.399836 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.399107    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 19 08:22:24.399836 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.399359    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 19 08:22:24.399836 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.399604    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 19 08:22:24.402715 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.402692    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-dpq85"]
Sep 19 08:22:24.402897 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.402883    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fplhj"]
Sep 19 08:22:24.432931 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.431726    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.436421 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.435454    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 19 08:22:24.436421 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.435703    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wn4fp\""
Sep 19 08:22:24.436421 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.436012    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 19 08:22:24.436421 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.436268    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 19 08:22:24.439875 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.439818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:24.523067 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.522921    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:24.569376 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/11edfda1-bde6-4329-b746-655d3a1cd773-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569406    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/11edfda1-bde6-4329-b746-655d3a1cd773-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61795267-8b75-45e6-8236-8478035beaed-sys\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569460    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61795267-8b75-45e6-8236-8478035beaed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569486    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569520    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11edfda1-bde6-4329-b746-655d3a1cd773-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569552    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxgg\" (UniqueName: \"kubernetes.io/projected/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-kube-api-access-qnxgg\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6k8m\" (UniqueName: \"kubernetes.io/projected/11edfda1-bde6-4329-b746-655d3a1cd773-kube-api-access-b6k8m\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61795267-8b75-45e6-8236-8478035beaed-node-exporter-wtmp\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11edfda1-bde6-4329-b746-655d3a1cd773-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/11edfda1-bde6-4329-b746-655d3a1cd773-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569790    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61795267-8b75-45e6-8236-8478035beaed-node-exporter-textfile\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.569815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61795267-8b75-45e6-8236-8478035beaed-metrics-client-ca\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.570409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.570409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61795267-8b75-45e6-8236-8478035beaed-node-exporter-tls\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.570409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569916    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61795267-8b75-45e6-8236-8478035beaed-node-exporter-accelerators-collector-config\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.570409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.569942    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7kd\" (UniqueName: \"kubernetes.io/projected/61795267-8b75-45e6-8236-8478035beaed-kube-api-access-xr7kd\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.570409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.570022    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.570409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.570064    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61795267-8b75-45e6-8236-8478035beaed-root\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.580547 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.580337    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-bqfng"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.672738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11edfda1-bde6-4329-b746-655d3a1cd773-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.672797    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/11edfda1-bde6-4329-b746-655d3a1cd773-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.672825    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61795267-8b75-45e6-8236-8478035beaed-node-exporter-textfile\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.672857    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61795267-8b75-45e6-8236-8478035beaed-metrics-client-ca\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.672891    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.672923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61795267-8b75-45e6-8236-8478035beaed-node-exporter-tls\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.672983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61795267-8b75-45e6-8236-8478035beaed-node-exporter-accelerators-collector-config\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673013    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7kd\" (UniqueName: \"kubernetes.io/projected/61795267-8b75-45e6-8236-8478035beaed-kube-api-access-xr7kd\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673081    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673109    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61795267-8b75-45e6-8236-8478035beaed-root\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/11edfda1-bde6-4329-b746-655d3a1cd773-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673197    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/11edfda1-bde6-4329-b746-655d3a1cd773-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61795267-8b75-45e6-8236-8478035beaed-sys\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673277    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61795267-8b75-45e6-8236-8478035beaed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.674282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11edfda1-bde6-4329-b746-655d3a1cd773-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.675178 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxgg\" (UniqueName: \"kubernetes.io/projected/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-kube-api-access-qnxgg\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.675178 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6k8m\" (UniqueName: \"kubernetes.io/projected/11edfda1-bde6-4329-b746-655d3a1cd773-kube-api-access-b6k8m\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.675178 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61795267-8b75-45e6-8236-8478035beaed-node-exporter-wtmp\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.675178 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.673640    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61795267-8b75-45e6-8236-8478035beaed-node-exporter-wtmp\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.675178 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.674698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11edfda1-bde6-4329-b746-655d3a1cd773-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.675178 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.675155    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61795267-8b75-45e6-8236-8478035beaed-root\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.676568 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.675525    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61795267-8b75-45e6-8236-8478035beaed-node-exporter-textfile\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.676568 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.675641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/11edfda1-bde6-4329-b746-655d3a1cd773-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.676568 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.675900    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.676568 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.676162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61795267-8b75-45e6-8236-8478035beaed-node-exporter-accelerators-collector-config\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.676568 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.676179    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/11edfda1-bde6-4329-b746-655d3a1cd773-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.676568 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.676225    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61795267-8b75-45e6-8236-8478035beaed-sys\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.681456 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.681431    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61795267-8b75-45e6-8236-8478035beaed-metrics-client-ca\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.684890 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.684840    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61795267-8b75-45e6-8236-8478035beaed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.693755 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.693721    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:24.697491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.697433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6k8m\" (UniqueName: \"kubernetes.io/projected/11edfda1-bde6-4329-b746-655d3a1cd773-kube-api-access-b6k8m\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.702676 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.702504    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7kd\" (UniqueName: \"kubernetes.io/projected/61795267-8b75-45e6-8236-8478035beaed-kube-api-access-xr7kd\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.706393 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.706135    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxgg\" (UniqueName: \"kubernetes.io/projected/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-kube-api-access-qnxgg\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.708612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.708564    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.710586 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.710536    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11edfda1-bde6-4329-b746-655d3a1cd773-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.711205 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.711167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/11edfda1-bde6-4329-b746-655d3a1cd773-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-dpq85\" (UID: \"11edfda1-bde6-4329-b746-655d3a1cd773\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:24.714021 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.713666    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-wxndl\" (UID: \"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:24.714021 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.713970    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61795267-8b75-45e6-8236-8478035beaed-node-exporter-tls\") pod \"node-exporter-fplhj\" (UID: \"61795267-8b75-45e6-8236-8478035beaed\") " pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.749060 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.748868    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fplhj"
Sep 19 08:22:24.788838 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.788727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:24.993190 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:24.992913    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"
Sep 19 08:22:25.008498 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.008464    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85"
Sep 19 08:22:25.105876 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:22:25.105836    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61795267_8b75_45e6_8236_8478035beaed.slice/crio-b6e2ffca95b0ccc71dbdfaee37572e90d716cb0511a42e811edcf543a305521f WatchSource:0}: Error finding container b6e2ffca95b0ccc71dbdfaee37572e90d716cb0511a42e811edcf543a305521f: Status 404 returned error can't find the container with id b6e2ffca95b0ccc71dbdfaee37572e90d716cb0511a42e811edcf543a305521f
Sep 19 08:22:25.107116 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.106999    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:25.210291 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.209418    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:25.325198 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.325090    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:25.401566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.401531    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-dpq85"]
Sep 19 08:22:25.410324 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.410295    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:22:25.436118 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.436084    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl"]
Sep 19 08:22:25.436277 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.436229    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.440558 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.440531    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 19 08:22:25.440819 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.440795    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 19 08:22:25.440886 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.440856    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 19 08:22:25.440886 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.440867    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9whrt\""
Sep 19 08:22:25.441157 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.441136    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 19 08:22:25.441310 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.441288    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 19 08:22:25.441367 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.441321    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 19 08:22:25.441577 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.441543    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 19 08:22:25.441690 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.441664    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 19 08:22:25.441742 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.441727    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 19 08:22:25.447635 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.447581    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:22:25.449740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.449713    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:25.476416 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:22:25.475587    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11edfda1_bde6_4329_b746_655d3a1cd773.slice/crio-85a8e91d0fe330eef67ba054d3dc09fb68923e20e1f19307eb868bc6be9c6d41 WatchSource:0}: Error finding container 85a8e91d0fe330eef67ba054d3dc09fb68923e20e1f19307eb868bc6be9c6d41: Status 404 returned error can't find the container with id 85a8e91d0fe330eef67ba054d3dc09fb68923e20e1f19307eb868bc6be9c6d41
Sep 19 08:22:25.477841 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:22:25.477531    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011ee6eb_d2a8_4c57_b205_2b3f31c2ad7f.slice/crio-7191445b383bc9ec586d9a51ee7eac31aa4517aeb1bfa15d33d83743fcf7ad39 WatchSource:0}: Error finding container 7191445b383bc9ec586d9a51ee7eac31aa4517aeb1bfa15d33d83743fcf7ad39: Status 404 returned error can't find the container with id 7191445b383bc9ec586d9a51ee7eac31aa4517aeb1bfa15d33d83743fcf7ad39
Sep 19 08:22:25.574389 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.573609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl" event={"ID":"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f","Type":"ContainerStarted","Data":"7191445b383bc9ec586d9a51ee7eac31aa4517aeb1bfa15d33d83743fcf7ad39"}
Sep 19 08:22:25.576001 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.575938    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fplhj" event={"ID":"61795267-8b75-45e6-8236-8478035beaed","Type":"ContainerStarted","Data":"b6e2ffca95b0ccc71dbdfaee37572e90d716cb0511a42e811edcf543a305521f"}
Sep 19 08:22:25.577617 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.577562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85" event={"ID":"11edfda1-bde6-4329-b746-655d3a1cd773","Type":"ContainerStarted","Data":"85a8e91d0fe330eef67ba054d3dc09fb68923e20e1f19307eb868bc6be9c6d41"}
Sep 19 08:22:25.582682 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.582661    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.582783 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.582708    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-web-config\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.582783 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.582754    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.582893 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.582800    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-config-out\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.582893 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.582843    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.582893 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.582877    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.583032 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.582905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-tls-assets\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.583032 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.582958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.583032 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.583017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.583230 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.583048    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsnz4\" (UniqueName: \"kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-kube-api-access-wsnz4\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.583230 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.583093    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.583379 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.583129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.583379 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.583334    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-config-volume\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-config-volume\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-web-config\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687467    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-config-out\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-tls-assets\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.687707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsnz4\" (UniqueName: \"kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-kube-api-access-wsnz4\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:22:25.688198    2559 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 19 08:22:25.688581 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:22:25.688284    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-main-tls podName:670b77fa-df60-4f24-adbd-01ae35baf479 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:26.188243826 +0000 UTC m=+98.598766817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479") : secret "alertmanager-main-tls" not found
Sep 19 08:22:25.690639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.690559    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.692267 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.691456    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.694017 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.693186    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.699323 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.698711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.699323 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.699286    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsnz4\" (UniqueName: \"kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-kube-api-access-wsnz4\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.699907 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.699861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.700762 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.700159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.700857 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.700773    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.701007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.700968    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-config-volume\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.701533 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.701492    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-tls-assets\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.701804 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.701767    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-config-out\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.703911 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.703875    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-web-config\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:25.732118 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.732046    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:25.832659 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:25.832556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:26.193839 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:26.193396    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:26.208352 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:26.208319    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:26.349667 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:26.349637    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:22:26.586934 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:26.586852    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t7vcg" event={"ID":"c64a1e78-7b83-4797-baca-6f3ccbea5a53","Type":"ContainerStarted","Data":"007c0b6a8f52d6d500d5d77f536ed0ce5a55ee2ec1501794c6ad8c1442c67c2e"}
Sep 19 08:22:26.590395 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:26.590349    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl" event={"ID":"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f","Type":"ContainerStarted","Data":"03a32d26818f3721f93c2201e521cad52cd5e8abea01a896dbf0e0e569dbcc95"}
Sep 19 08:22:26.590550 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:26.590421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl" event={"ID":"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f","Type":"ContainerStarted","Data":"b4ba087b68c226a1524845c770740177c803002c10681fd8bdaba91fe5fffa88"}
Sep 19 08:22:26.606794 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:26.606736    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t7vcg" podStartSLOduration=34.6098554 podStartE2EDuration="44.606713503s" totalImagesPullingTime="9.996858103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:21:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:26.604147255 +0000 UTC m=+99.014670259" watchObservedRunningTime="2026-09-19 08:22:26.606713503 +0000 UTC m=+99.017236504"
Sep 19 08:22:27.220581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:27.220448    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:27.327168 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:27.327136    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:27.433748 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:27.433711    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:27.484525 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:27.484485    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:27.864964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:27.860599    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:22:27.867455 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:22:27.867418    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670b77fa_df60_4f24_adbd_01ae35baf479.slice/crio-3664d0a36bb56772d4e98dfe8ca5229e657b32dc8c0be5c3fba4592e25e0d8bb WatchSource:0}: Error finding container 3664d0a36bb56772d4e98dfe8ca5229e657b32dc8c0be5c3fba4592e25e0d8bb: Status 404 returned error can't find the container with id 3664d0a36bb56772d4e98dfe8ca5229e657b32dc8c0be5c3fba4592e25e0d8bb
Sep 19 08:22:28.100591 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.100556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:28.500668 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.500569    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:28.503836 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.503794    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c6758476b-lz964"
Sep 19 08:22:28.579825 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.579734    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:28.604508 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.604405    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerStarted","Data":"3664d0a36bb56772d4e98dfe8ca5229e657b32dc8c0be5c3fba4592e25e0d8bb"}
Sep 19 08:22:28.609515 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.609274    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl" event={"ID":"011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f","Type":"ContainerStarted","Data":"b2b75ad1a9443c1602f248eaf6f294235db3523bdbc0e041b60c204665633f4b"}
Sep 19 08:22:28.611971 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.611940    2559 generic.go:356] "Generic (PLEG): container finished" podID="61795267-8b75-45e6-8236-8478035beaed" containerID="1ea4d68e23cb8bcb2378161dc1606eef7ed421654501b2ac395550d294702885" exitCode=0
Sep 19 08:22:28.612197 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.612041    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fplhj" event={"ID":"61795267-8b75-45e6-8236-8478035beaed","Type":"ContainerDied","Data":"1ea4d68e23cb8bcb2378161dc1606eef7ed421654501b2ac395550d294702885"}
Sep 19 08:22:28.616587 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.615910    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85" event={"ID":"11edfda1-bde6-4329-b746-655d3a1cd773","Type":"ContainerStarted","Data":"bacd6c43dd4ec0662b6d683155c3afba6115a5a4f3f8bed0788d1b8a4075c3b6"}
Sep 19 08:22:28.616587 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.615987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85" event={"ID":"11edfda1-bde6-4329-b746-655d3a1cd773","Type":"ContainerStarted","Data":"c5a92d1d496815b05c1408a681174f318ad0a55343d4df4c2935cb9d443857ed"}
Sep 19 08:22:28.616587 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.616018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85" event={"ID":"11edfda1-bde6-4329-b746-655d3a1cd773","Type":"ContainerStarted","Data":"6f3d157db2ed232551285acad4227688e518bb0ed8129bdddd57c122be22dfe6"}
Sep 19 08:22:28.631582 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.631396    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-wxndl" podStartSLOduration=2.534688639 podStartE2EDuration="4.631375548s" totalImagesPullingTime="2.096686909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:28.63018385 +0000 UTC m=+101.040706850" watchObservedRunningTime="2026-09-19 08:22:28.631375548 +0000 UTC m=+101.041898548"
Sep 19 08:22:28.651838 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.651805    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:28.751785 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:28.751676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:29.033532 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:29.033492    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:29.108202 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:29.108164    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:29.168869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:29.168829    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:29.355470 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:29.355386    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:29.447397 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:29.447368    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:29.623939 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:29.623841    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fplhj" event={"ID":"61795267-8b75-45e6-8236-8478035beaed","Type":"ContainerStarted","Data":"0ddc6ec5e266e71cde74d3538ff00a61b84e6797738b0f615b7ec4a9b88fa5f7"}
Sep 19 08:22:29.624136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:29.623951    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fplhj" event={"ID":"61795267-8b75-45e6-8236-8478035beaed","Type":"ContainerStarted","Data":"b96224fa7bafe21cc0b27dc8b1a59e3a51e334f4c262a535478846503187b38a"}
Sep 19 08:22:29.645517 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:29.645457    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fplhj" podStartSLOduration=3.089959856 podStartE2EDuration="5.645433979s" totalImagesPullingTime="2.448927271s" totalInitContainerRuntime="106.546852ms" isStatefulPod=true podCreationTimestamp="2026-09-19 08:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:29.644094264 +0000 UTC m=+102.054617265" watchObservedRunningTime="2026-09-19 08:22:29.645433979 +0000 UTC m=+102.055956980"
Sep 19 08:22:29.645772 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:29.645733    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-dpq85" podStartSLOduration=3.589798632 podStartE2EDuration="5.645724962s" totalImagesPullingTime="2.05592633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:28.677309378 +0000 UTC m=+101.087832379" watchObservedRunningTime="2026-09-19 08:22:29.645724962 +0000 UTC m=+102.056247962"
Sep 19 08:22:30.518744 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.518701    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:30.633020 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.632984    2559 generic.go:356] "Generic (PLEG): container finished" podID="670b77fa-df60-4f24-adbd-01ae35baf479" containerID="841eef7d41756c5c11353d6d2abcaac52b4c499c9745ad2fc1795bac20d97a07" exitCode=0
Sep 19 08:22:30.633578 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.633539    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerDied","Data":"841eef7d41756c5c11353d6d2abcaac52b4c499c9745ad2fc1795bac20d97a07"}
Sep 19 08:22:30.816052 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.816000    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp" podUID="b5bace44-66ba-4d16-8634-a518af242b6c" containerName="registry" containerID="cri-o://64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7" gracePeriod=30
Sep 19 08:22:30.955529 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.955495    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54b64d5486-wlnnw"]
Sep 19 08:22:30.982343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.975784    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54b64d5486-wlnnw"]
Sep 19 08:22:30.982343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.975900    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:30.982343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.979164    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 19 08:22:30.982343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.979388    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 19 08:22:30.982343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.979809    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 19 08:22:30.982343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.979846    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 19 08:22:30.982343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.980639    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 19 08:22:30.982343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.980831    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7kqxm\""
Sep 19 08:22:30.993383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:30.984965    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 19 08:22:31.041152 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.041114    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-oauth-serving-cert\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.041356 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.041205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-serving-cert\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.041356 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.041283    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-oauth-config\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.041356 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.041346    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-trusted-ca-bundle\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.041541 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.041389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-service-ca\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.041541 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.041443    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-console-config\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.041541 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.041468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84zp6\" (UniqueName: \"kubernetes.io/projected/b5f0ad13-9117-4334-b675-3d7973f1c219-kube-api-access-84zp6\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.117713 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.117679    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:22:31.142331 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142288    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-registry-certificates\") pod \"b5bace44-66ba-4d16-8634-a518af242b6c\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") "
Sep 19 08:22:31.142767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142352    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-trusted-ca\") pod \"b5bace44-66ba-4d16-8634-a518af242b6c\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") "
Sep 19 08:22:31.142767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142383    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-installation-pull-secrets\") pod \"b5bace44-66ba-4d16-8634-a518af242b6c\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") "
Sep 19 08:22:31.142767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142424    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-image-registry-private-configuration\") pod \"b5bace44-66ba-4d16-8634-a518af242b6c\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") "
Sep 19 08:22:31.142767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142462    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5bace44-66ba-4d16-8634-a518af242b6c-ca-trust-extracted\") pod \"b5bace44-66ba-4d16-8634-a518af242b6c\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") "
Sep 19 08:22:31.142767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142482    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-bound-sa-token\") pod \"b5bace44-66ba-4d16-8634-a518af242b6c\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") "
Sep 19 08:22:31.142767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142557    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls\") pod \"b5bace44-66ba-4d16-8634-a518af242b6c\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") "
Sep 19 08:22:31.142767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142599    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc5h7\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-kube-api-access-mc5h7\") pod \"b5bace44-66ba-4d16-8634-a518af242b6c\" (UID: \"b5bace44-66ba-4d16-8634-a518af242b6c\") "
Sep 19 08:22:31.142767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142697    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zp6\" (UniqueName: \"kubernetes.io/projected/b5f0ad13-9117-4334-b675-3d7973f1c219-kube-api-access-84zp6\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.142767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142714    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b5bace44-66ba-4d16-8634-a518af242b6c" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:31.142767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-oauth-serving-cert\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-serving-cert\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-oauth-config\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-trusted-ca-bundle\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-service-ca\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.142992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-console-config\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.143001    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b5bace44-66ba-4d16-8634-a518af242b6c" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.143041    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-registry-certificates\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.144114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-oauth-serving-cert\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.147018    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-trusted-ca-bundle\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.147577    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-service-ca\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.148018    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-kube-api-access-mc5h7" (OuterVolumeSpecName: "kube-api-access-mc5h7") pod "b5bace44-66ba-4d16-8634-a518af242b6c" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c"). InnerVolumeSpecName "kube-api-access-mc5h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.148193    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-console-config\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.149087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.148368    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-oauth-config\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.150495 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.150449    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b5bace44-66ba-4d16-8634-a518af242b6c" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:22:31.152141 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.151998    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b5bace44-66ba-4d16-8634-a518af242b6c" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:22:31.152141 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.152045    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b5bace44-66ba-4d16-8634-a518af242b6c" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:22:31.152335 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.152204    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b5bace44-66ba-4d16-8634-a518af242b6c" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:22:31.157384 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.157356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zp6\" (UniqueName: \"kubernetes.io/projected/b5f0ad13-9117-4334-b675-3d7973f1c219-kube-api-access-84zp6\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.160145 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.160121    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-serving-cert\") pod \"console-54b64d5486-wlnnw\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") " pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.162955 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.162847    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bace44-66ba-4d16-8634-a518af242b6c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b5bace44-66ba-4d16-8634-a518af242b6c" (UID: "b5bace44-66ba-4d16-8634-a518af242b6c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:22:31.244409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.244376    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5bace44-66ba-4d16-8634-a518af242b6c-trusted-ca\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:22:31.244409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.244404    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-installation-pull-secrets\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:22:31.244409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.244417    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5bace44-66ba-4d16-8634-a518af242b6c-image-registry-private-configuration\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:22:31.244695 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.244430    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5bace44-66ba-4d16-8634-a518af242b6c-ca-trust-extracted\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:22:31.244695 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.244444    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-bound-sa-token\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:22:31.244695 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.244456    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-registry-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:22:31.244695 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.244468    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc5h7\" (UniqueName: \"kubernetes.io/projected/b5bace44-66ba-4d16-8634-a518af242b6c-kube-api-access-mc5h7\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:22:31.288786 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.288754    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:31.467892 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.467855    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gcm28"
Sep 19 08:22:31.605779 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.605532    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54b64d5486-wlnnw"]
Sep 19 08:22:31.608825 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:22:31.608790    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f0ad13_9117_4334_b675_3d7973f1c219.slice/crio-48f376c6f46266e6a8e8fd19f666d8267ccdfcba61a08a02bfe03a9a92bdd199 WatchSource:0}: Error finding container 48f376c6f46266e6a8e8fd19f666d8267ccdfcba61a08a02bfe03a9a92bdd199: Status 404 returned error can't find the container with id 48f376c6f46266e6a8e8fd19f666d8267ccdfcba61a08a02bfe03a9a92bdd199
Sep 19 08:22:31.640690 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.640567    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b64d5486-wlnnw" event={"ID":"b5f0ad13-9117-4334-b675-3d7973f1c219","Type":"ContainerStarted","Data":"48f376c6f46266e6a8e8fd19f666d8267ccdfcba61a08a02bfe03a9a92bdd199"}
Sep 19 08:22:31.642400 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.642366    2559 generic.go:356] "Generic (PLEG): container finished" podID="b5bace44-66ba-4d16-8634-a518af242b6c" containerID="64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7" exitCode=0
Sep 19 08:22:31.642533 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.642430    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp" event={"ID":"b5bace44-66ba-4d16-8634-a518af242b6c","Type":"ContainerDied","Data":"64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7"}
Sep 19 08:22:31.642533 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.642465    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"
Sep 19 08:22:31.642533 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.642479    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcfbdb9c6-wc2zp" event={"ID":"b5bace44-66ba-4d16-8634-a518af242b6c","Type":"ContainerDied","Data":"b97c9978200f8af146fca9fbe3de54c1f3634fb20eaf148e7909d41b3770eae1"}
Sep 19 08:22:31.642533 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.642509    2559 scope.go:117] "RemoveContainer" containerID="64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7"
Sep 19 08:22:31.657952 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.657925    2559 scope.go:117] "RemoveContainer" containerID="64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7"
Sep 19 08:22:31.658376 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:22:31.658345    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7\": container with ID starting with 64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7 not found: ID does not exist" containerID="64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7"
Sep 19 08:22:31.658504 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.658383    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7"} err="failed to get container status \"64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7\": rpc error: code = NotFound desc = could not find container \"64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7\": container with ID starting with 64470557f3734dd0da208c70e40768acee7e91f67d34b8715b8ee0a760a15cc7 not found: ID does not exist"
Sep 19 08:22:31.668967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.668930    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"]
Sep 19 08:22:31.671863 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.671840    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bcfbdb9c6-wc2zp"]
Sep 19 08:22:31.939705 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:31.939617    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5bace44-66ba-4d16-8634-a518af242b6c" path="/var/lib/kubelet/pods/b5bace44-66ba-4d16-8634-a518af242b6c/volumes"
Sep 19 08:22:33.657590 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:33.657515    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerStarted","Data":"3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b"}
Sep 19 08:22:33.657590 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:33.657608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerStarted","Data":"b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4"}
Sep 19 08:22:33.658072 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:33.657651    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerStarted","Data":"c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4"}
Sep 19 08:22:33.658072 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:33.657694    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerStarted","Data":"f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d"}
Sep 19 08:22:33.658072 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:33.657735    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerStarted","Data":"fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5"}
Sep 19 08:22:34.749796 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:34.749758    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:34.819167 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:34.819112    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:37.032494 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:37.032460    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:37.099894 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:37.099863    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:37.677415 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:37.677383    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerStarted","Data":"bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3"}
Sep 19 08:22:37.679078 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:37.679051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b64d5486-wlnnw" event={"ID":"b5f0ad13-9117-4334-b675-3d7973f1c219","Type":"ContainerStarted","Data":"1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1"}
Sep 19 08:22:37.707595 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:37.707546    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.893608494 podStartE2EDuration="12.707530182s" totalImagesPullingTime="7.795559248s" totalInitContainerRuntime="18.36244ms" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:37.704841327 +0000 UTC m=+110.115364328" watchObservedRunningTime="2026-09-19 08:22:37.707530182 +0000 UTC m=+110.118053181"
Sep 19 08:22:37.725485 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:37.725243    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54b64d5486-wlnnw" podStartSLOduration=2.143332893 podStartE2EDuration="7.725221908s" totalImagesPullingTime="5.581889015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:37.724873024 +0000 UTC m=+110.135396025" watchObservedRunningTime="2026-09-19 08:22:37.725221908 +0000 UTC m=+110.135744907"
Sep 19 08:22:41.118041 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:41.118007    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:41.170583 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:41.170548    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:41.289901 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:41.289858    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:41.290081 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:41.289932    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:41.294714 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:41.294691    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:41.697788 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:41.697760    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:22:41.852685 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:41.852653    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:41.952316 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:41.952234    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:42.009436 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:42.009405    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:48.482313 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:48.482278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:48.585247 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:48.585200    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:49.742300 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:49.742239    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:49.844332 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:49.844299    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:22:58.810063 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:58.810025    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:22:58.812439 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:58.812416    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9178e60e-1f54-461c-8a40-1b8c3300b64c-metrics-certs\") pod \"network-metrics-daemon-7qn5v\" (UID: \"9178e60e-1f54-461c-8a40-1b8c3300b64c\") " pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:22:58.854421 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:58.854388    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c9bjw\""
Sep 19 08:22:58.861720 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:58.861700    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qn5v"
Sep 19 08:22:59.167457 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:59.167379    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qn5v"]
Sep 19 08:22:59.170670 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:22:59.170641    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9178e60e_1f54_461c_8a40_1b8c3300b64c.slice/crio-2ffb9dc163d9725dcd1517459764938548d674e8f9892d9c5991edf629a77a8b WatchSource:0}: Error finding container 2ffb9dc163d9725dcd1517459764938548d674e8f9892d9c5991edf629a77a8b: Status 404 returned error can't find the container with id 2ffb9dc163d9725dcd1517459764938548d674e8f9892d9c5991edf629a77a8b
Sep 19 08:22:59.765581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:22:59.765528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qn5v" event={"ID":"9178e60e-1f54-461c-8a40-1b8c3300b64c","Type":"ContainerStarted","Data":"2ffb9dc163d9725dcd1517459764938548d674e8f9892d9c5991edf629a77a8b"}
Sep 19 08:23:00.771385 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:00.771287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qn5v" event={"ID":"9178e60e-1f54-461c-8a40-1b8c3300b64c","Type":"ContainerStarted","Data":"57ce8f4c164df2799a633a6a7d20c7ff06b79c0a0fc5060ad751ff00c3cf20c9"}
Sep 19 08:23:00.771385 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:00.771354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qn5v" event={"ID":"9178e60e-1f54-461c-8a40-1b8c3300b64c","Type":"ContainerStarted","Data":"de43e6ce7bab57cd0962820c6393daf3aa1ea2fb0297470cae1343ef313559b6"}
Sep 19 08:23:00.789119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:00.789070    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7qn5v" podStartSLOduration=129.564154686 podStartE2EDuration="2m10.789050036s" totalImagesPullingTime="1.22489535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:23:00.788640346 +0000 UTC m=+133.199163360" watchObservedRunningTime="2026-09-19 08:23:00.789050036 +0000 UTC m=+133.199573038"
Sep 19 08:23:00.879800 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:00.879762    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:23:00.979639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:00.979607    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:23:04.741771 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:04.741729    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:23:04.800467 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:04.800425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:23:07.798703 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:07.798666    2559 generic.go:356] "Generic (PLEG): container finished" podID="af453c92-08c7-44c0-9e10-52cf98b6b7f4" containerID="63fe1d343358a6a2ab4dca20f602604df95239193241c6758699d5a724727762" exitCode=0
Sep 19 08:23:07.799170 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:07.798737    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64" event={"ID":"af453c92-08c7-44c0-9e10-52cf98b6b7f4","Type":"ContainerDied","Data":"63fe1d343358a6a2ab4dca20f602604df95239193241c6758699d5a724727762"}
Sep 19 08:23:07.799170 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:07.799049    2559 scope.go:117] "RemoveContainer" containerID="63fe1d343358a6a2ab4dca20f602604df95239193241c6758699d5a724727762"
Sep 19 08:23:08.807029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:08.806991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-k7g64" event={"ID":"af453c92-08c7-44c0-9e10-52cf98b6b7f4","Type":"ContainerStarted","Data":"1de05f11842fd27f7e7ed6a0026fe307059e0ceb892da189627f9383e8ed2296"}
Sep 19 08:23:11.109342 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:11.109297    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:23:11.820888 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:11.820855    2559 generic.go:356] "Generic (PLEG): container finished" podID="60dc08bc-8f96-458d-a5a7-c09534ca4ef5" containerID="1c2e45a5c172bc8aa66e4ebda1e78e3bb3b12f09dcb02083661cc8ca49dafba4" exitCode=0
Sep 19 08:23:11.821092 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:11.820907    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-hhscx" event={"ID":"60dc08bc-8f96-458d-a5a7-c09534ca4ef5","Type":"ContainerDied","Data":"1c2e45a5c172bc8aa66e4ebda1e78e3bb3b12f09dcb02083661cc8ca49dafba4"}
Sep 19 08:23:11.821329 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:11.821311    2559 scope.go:117] "RemoveContainer" containerID="1c2e45a5c172bc8aa66e4ebda1e78e3bb3b12f09dcb02083661cc8ca49dafba4"
Sep 19 08:23:12.828427 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:12.828388    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-hhscx" event={"ID":"60dc08bc-8f96-458d-a5a7-c09534ca4ef5","Type":"ContainerStarted","Data":"c110ab51b68cad2b109c315e711ddb0aaa2016b3d4ed4537ae2316ec14c888ae"}
Sep 19 08:23:43.474535 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:43.474447    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:23:43.675483 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:43.675448    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:23:44.534195 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.534158    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:23:44.534603 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.534505    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="alertmanager" containerID="cri-o://fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5" gracePeriod=120
Sep 19 08:23:44.534603 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.534558    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy-metric" containerID="cri-o://3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b" gracePeriod=120
Sep 19 08:23:44.534729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.534602    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="config-reloader" containerID="cri-o://f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d" gracePeriod=120
Sep 19 08:23:44.534729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.534591    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy-web" containerID="cri-o://c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4" gracePeriod=120
Sep 19 08:23:44.534729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.534646    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy" containerID="cri-o://b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4" gracePeriod=120
Sep 19 08:23:44.534729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.534685    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="prom-label-proxy" containerID="cri-o://bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3" gracePeriod=120
Sep 19 08:23:44.959438 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.959358    2559 generic.go:356] "Generic (PLEG): container finished" podID="670b77fa-df60-4f24-adbd-01ae35baf479" containerID="bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3" exitCode=0
Sep 19 08:23:44.959438 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.959382    2559 generic.go:356] "Generic (PLEG): container finished" podID="670b77fa-df60-4f24-adbd-01ae35baf479" containerID="3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b" exitCode=0
Sep 19 08:23:44.959438 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.959389    2559 generic.go:356] "Generic (PLEG): container finished" podID="670b77fa-df60-4f24-adbd-01ae35baf479" containerID="b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4" exitCode=0
Sep 19 08:23:44.959438 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.959395    2559 generic.go:356] "Generic (PLEG): container finished" podID="670b77fa-df60-4f24-adbd-01ae35baf479" containerID="f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d" exitCode=0
Sep 19 08:23:44.959438 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.959400    2559 generic.go:356] "Generic (PLEG): container finished" podID="670b77fa-df60-4f24-adbd-01ae35baf479" containerID="fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5" exitCode=0
Sep 19 08:23:44.959438 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.959426    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerDied","Data":"bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3"}
Sep 19 08:23:44.959725 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.959490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerDied","Data":"3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b"}
Sep 19 08:23:44.959725 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.959516    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerDied","Data":"b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4"}
Sep 19 08:23:44.959725 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.959544    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerDied","Data":"f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d"}
Sep 19 08:23:44.959725 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:44.959567    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerDied","Data":"fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5"}
Sep 19 08:23:45.778985 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.778958    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:45.878498 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878410    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-tls-assets\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.878498 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878453    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-metrics-client-ca\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.878498 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878476    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-config-out\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.878731 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878509    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-metric\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.878731 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878543    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-web-config\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.878731 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878594    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-cluster-tls-config\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.878731 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878644    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-trusted-ca-bundle\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.878731 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878677    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-main-tls\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.878731 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878720    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-main-db\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.879029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878749    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-config-volume\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.879029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878780    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.879029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878807    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsnz4\" (UniqueName: \"kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-kube-api-access-wsnz4\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.879029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878837    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-web\") pod \"670b77fa-df60-4f24-adbd-01ae35baf479\" (UID: \"670b77fa-df60-4f24-adbd-01ae35baf479\") "
Sep 19 08:23:45.879029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.878945    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:45.879297 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.879058    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:23:45.879297 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.879118    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-metrics-client-ca\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.879759 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.879726    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:23:45.881951 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.881890    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:23:45.881951 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.881913    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:45.882131 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.882104    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:45.882332 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.882298    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-config-out" (OuterVolumeSpecName: "config-out") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:23:45.882465 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.882442    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:45.882639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.882613    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-config-volume" (OuterVolumeSpecName: "config-volume") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:45.882937 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.882917    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:45.883399 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.883381    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-kube-api-access-wsnz4" (OuterVolumeSpecName: "kube-api-access-wsnz4") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "kube-api-access-wsnz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:23:45.886290 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.886267    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:45.892667 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.892645    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-web-config" (OuterVolumeSpecName: "web-config") pod "670b77fa-df60-4f24-adbd-01ae35baf479" (UID: "670b77fa-df60-4f24-adbd-01ae35baf479"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:23:45.966735 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.966700    2559 generic.go:356] "Generic (PLEG): container finished" podID="670b77fa-df60-4f24-adbd-01ae35baf479" containerID="c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4" exitCode=0
Sep 19 08:23:45.966896 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.966746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerDied","Data":"c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4"}
Sep 19 08:23:45.966896 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.966784    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:45.966896 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.966839    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"670b77fa-df60-4f24-adbd-01ae35baf479","Type":"ContainerDied","Data":"3664d0a36bb56772d4e98dfe8ca5229e657b32dc8c0be5c3fba4592e25e0d8bb"}
Sep 19 08:23:45.966896 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.966886    2559 scope.go:117] "RemoveContainer" containerID="bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3"
Sep 19 08:23:45.974708 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.974686    2559 scope.go:117] "RemoveContainer" containerID="3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b"
Sep 19 08:23:45.979977 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.979953    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.979977 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.979972    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsnz4\" (UniqueName: \"kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-kube-api-access-wsnz4\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.980136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.979984    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.980136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.979994    2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/670b77fa-df60-4f24-adbd-01ae35baf479-tls-assets\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.980136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.980003    2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-config-out\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.980136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.980011    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.980136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.980019    2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-web-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.980136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.980027    2559 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-cluster-tls-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.980136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.980036    2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.980136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.980044    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-secret-alertmanager-main-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.980136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.980052    2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/670b77fa-df60-4f24-adbd-01ae35baf479-alertmanager-main-db\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.980136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.980060    2559 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/670b77fa-df60-4f24-adbd-01ae35baf479-config-volume\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:23:45.981797 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.981784    2559 scope.go:117] "RemoveContainer" containerID="b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4"
Sep 19 08:23:45.984740 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.984720    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:23:45.989483 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.989462    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:23:45.990679 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.990664    2559 scope.go:117] "RemoveContainer" containerID="c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4"
Sep 19 08:23:45.997268 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:45.997232    2559 scope.go:117] "RemoveContainer" containerID="f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d"
Sep 19 08:23:46.004071 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.004052    2559 scope.go:117] "RemoveContainer" containerID="fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5"
Sep 19 08:23:46.010479 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010456    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:23:46.010851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010836    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5bace44-66ba-4d16-8634-a518af242b6c" containerName="registry"
Sep 19 08:23:46.010895 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010855    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5bace44-66ba-4d16-8634-a518af242b6c" containerName="registry"
Sep 19 08:23:46.010895 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010868    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy-metric"
Sep 19 08:23:46.010895 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010876    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy-metric"
Sep 19 08:23:46.010895 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010891    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="alertmanager"
Sep 19 08:23:46.011007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010899    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="alertmanager"
Sep 19 08:23:46.011007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010911    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy-web"
Sep 19 08:23:46.011007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010919    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy-web"
Sep 19 08:23:46.011007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010929    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="init-config-reloader"
Sep 19 08:23:46.011007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010937    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="init-config-reloader"
Sep 19 08:23:46.011007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010960    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="prom-label-proxy"
Sep 19 08:23:46.011007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010971    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="prom-label-proxy"
Sep 19 08:23:46.011007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010989    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="config-reloader"
Sep 19 08:23:46.011007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.010997    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="config-reloader"
Sep 19 08:23:46.011340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.011012    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy"
Sep 19 08:23:46.011340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.011021    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy"
Sep 19 08:23:46.011340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.011103    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy"
Sep 19 08:23:46.011340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.011117    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="alertmanager"
Sep 19 08:23:46.011340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.011126    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="config-reloader"
Sep 19 08:23:46.011340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.011135    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy-metric"
Sep 19 08:23:46.011340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.011144    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="prom-label-proxy"
Sep 19 08:23:46.011340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.011154    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5bace44-66ba-4d16-8634-a518af242b6c" containerName="registry"
Sep 19 08:23:46.011340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.011164    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" containerName="kube-rbac-proxy-web"
Sep 19 08:23:46.012441 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.012424    2559 scope.go:117] "RemoveContainer" containerID="841eef7d41756c5c11353d6d2abcaac52b4c499c9745ad2fc1795bac20d97a07"
Sep 19 08:23:46.014940 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.014919    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.017410 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.017387    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 19 08:23:46.017506 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.017419    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 19 08:23:46.017562 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.017390    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 19 08:23:46.017562 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.017533    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 19 08:23:46.017665 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.017590    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 19 08:23:46.017665 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.017647    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 19 08:23:46.018039 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.017856    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 19 08:23:46.018039 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.017924    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9whrt\""
Sep 19 08:23:46.018039 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.017994    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 19 08:23:46.019670 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.019650    2559 scope.go:117] "RemoveContainer" containerID="bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3"
Sep 19 08:23:46.019971 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:23:46.019946    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3\": container with ID starting with bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3 not found: ID does not exist" containerID="bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3"
Sep 19 08:23:46.020073 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.019982    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3"} err="failed to get container status \"bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3\": rpc error: code = NotFound desc = could not find container \"bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3\": container with ID starting with bc54c947478663f870148ab43ea9804866636c67fdc47c9c33721585834a37e3 not found: ID does not exist"
Sep 19 08:23:46.020073 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.020006    2559 scope.go:117] "RemoveContainer" containerID="3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b"
Sep 19 08:23:46.020291 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:23:46.020269    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b\": container with ID starting with 3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b not found: ID does not exist" containerID="3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b"
Sep 19 08:23:46.020376 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.020295    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b"} err="failed to get container status \"3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b\": rpc error: code = NotFound desc = could not find container \"3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b\": container with ID starting with 3d611b44935db19c7546fc51b02f5d60353193a1aa2dc4abd38cd768f737a26b not found: ID does not exist"
Sep 19 08:23:46.020376 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.020311    2559 scope.go:117] "RemoveContainer" containerID="b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4"
Sep 19 08:23:46.020574 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:23:46.020557    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4\": container with ID starting with b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4 not found: ID does not exist" containerID="b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4"
Sep 19 08:23:46.020617 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.020580    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4"} err="failed to get container status \"b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4\": rpc error: code = NotFound desc = could not find container \"b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4\": container with ID starting with b5cc6bd6bc59f93d8a2790d642e9004fce1591b95073571afc4d3a9e3c00cbb4 not found: ID does not exist"
Sep 19 08:23:46.020655 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.020616    2559 scope.go:117] "RemoveContainer" containerID="c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4"
Sep 19 08:23:46.020878 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:23:46.020860    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4\": container with ID starting with c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4 not found: ID does not exist" containerID="c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4"
Sep 19 08:23:46.020968 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.020878    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4"} err="failed to get container status \"c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4\": rpc error: code = NotFound desc = could not find container \"c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4\": container with ID starting with c06ed5c6e245ffc8a94ac354c62ad371ed0dd0a7f73007fcdbd99813f8d713a4 not found: ID does not exist"
Sep 19 08:23:46.020968 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.020890    2559 scope.go:117] "RemoveContainer" containerID="f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d"
Sep 19 08:23:46.021341 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:23:46.021313    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d\": container with ID starting with f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d not found: ID does not exist" containerID="f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d"
Sep 19 08:23:46.021418 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.021349    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d"} err="failed to get container status \"f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d\": rpc error: code = NotFound desc = could not find container \"f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d\": container with ID starting with f2176740177df57ec5c7c619c9182349a2cf6d5891dd8146c923882d1535d27d not found: ID does not exist"
Sep 19 08:23:46.021418 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.021370    2559 scope.go:117] "RemoveContainer" containerID="fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5"
Sep 19 08:23:46.021637 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:23:46.021619    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5\": container with ID starting with fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5 not found: ID does not exist" containerID="fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5"
Sep 19 08:23:46.021701 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.021644    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5"} err="failed to get container status \"fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5\": rpc error: code = NotFound desc = could not find container \"fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5\": container with ID starting with fe9483ee075f9b9a6fa664d725a92f2b3d82964f0c98ca1d6b8a9eb2812bdcc5 not found: ID does not exist"
Sep 19 08:23:46.021701 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.021664    2559 scope.go:117] "RemoveContainer" containerID="841eef7d41756c5c11353d6d2abcaac52b4c499c9745ad2fc1795bac20d97a07"
Sep 19 08:23:46.021896 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:23:46.021878    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841eef7d41756c5c11353d6d2abcaac52b4c499c9745ad2fc1795bac20d97a07\": container with ID starting with 841eef7d41756c5c11353d6d2abcaac52b4c499c9745ad2fc1795bac20d97a07 not found: ID does not exist" containerID="841eef7d41756c5c11353d6d2abcaac52b4c499c9745ad2fc1795bac20d97a07"
Sep 19 08:23:46.021952 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.021903    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841eef7d41756c5c11353d6d2abcaac52b4c499c9745ad2fc1795bac20d97a07"} err="failed to get container status \"841eef7d41756c5c11353d6d2abcaac52b4c499c9745ad2fc1795bac20d97a07\": rpc error: code = NotFound desc = could not find container \"841eef7d41756c5c11353d6d2abcaac52b4c499c9745ad2fc1795bac20d97a07\": container with ID starting with 841eef7d41756c5c11353d6d2abcaac52b4c499c9745ad2fc1795bac20d97a07 not found: ID does not exist"
Sep 19 08:23:46.022951 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.022935    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 19 08:23:46.029612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.029591    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:23:46.081073 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081073 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8aa0250-06dd-4d6f-b4dc-880851872780-config-out\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081137    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081155    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-web-config\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081171    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vj4\" (UniqueName: \"kubernetes.io/projected/d8aa0250-06dd-4d6f-b4dc-880851872780-kube-api-access-68vj4\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081199    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d8aa0250-06dd-4d6f-b4dc-880851872780-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081249    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa0250-06dd-4d6f-b4dc-880851872780-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081521 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081521 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8aa0250-06dd-4d6f-b4dc-880851872780-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081521 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-config-volume\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081521 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081417    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8aa0250-06dd-4d6f-b4dc-880851872780-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.081521 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.081454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.182887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.182788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-config-volume\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.182887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.182827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8aa0250-06dd-4d6f-b4dc-880851872780-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.182887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.182856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.182887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.182885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.183282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.182913    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8aa0250-06dd-4d6f-b4dc-880851872780-config-out\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.183282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.182954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.183282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.182998    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-web-config\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.183282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.183036    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vj4\" (UniqueName: \"kubernetes.io/projected/d8aa0250-06dd-4d6f-b4dc-880851872780-kube-api-access-68vj4\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.183282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.183084    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.183282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.183126    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d8aa0250-06dd-4d6f-b4dc-880851872780-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.183282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.183171    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa0250-06dd-4d6f-b4dc-880851872780-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.183282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.183215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.183282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.183241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8aa0250-06dd-4d6f-b4dc-880851872780-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.183966 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.183922    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d8aa0250-06dd-4d6f-b4dc-880851872780-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.184581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.184473    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8aa0250-06dd-4d6f-b4dc-880851872780-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.184684 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.184654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa0250-06dd-4d6f-b4dc-880851872780-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.185820 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.185776    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8aa0250-06dd-4d6f-b4dc-880851872780-config-out\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.186120 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.186089    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-config-volume\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.186208 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.186169    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.186208 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.186171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8aa0250-06dd-4d6f-b4dc-880851872780-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.186569 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.186546    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-web-config\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.186830 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.186805    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.187141 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.187122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.187141 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.187135    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.187502 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.187486    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d8aa0250-06dd-4d6f-b4dc-880851872780-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.192320 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.192301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vj4\" (UniqueName: \"kubernetes.io/projected/d8aa0250-06dd-4d6f-b4dc-880851872780-kube-api-access-68vj4\") pod \"alertmanager-main-0\" (UID: \"d8aa0250-06dd-4d6f-b4dc-880851872780\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.325755 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.325718    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 19 08:23:46.605429 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.605403    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 19 08:23:46.606568 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:23:46.606540    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8aa0250_06dd_4d6f_b4dc_880851872780.slice/crio-480542c85bf424a4353e7d90f0c5c5cc9bb95941c42c0fcb6c3c0105e6d1b2ac WatchSource:0}: Error finding container 480542c85bf424a4353e7d90f0c5c5cc9bb95941c42c0fcb6c3c0105e6d1b2ac: Status 404 returned error can't find the container with id 480542c85bf424a4353e7d90f0c5c5cc9bb95941c42c0fcb6c3c0105e6d1b2ac
Sep 19 08:23:46.977863 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.977768    2559 generic.go:356] "Generic (PLEG): container finished" podID="d8aa0250-06dd-4d6f-b4dc-880851872780" containerID="7affdec82c941242d6d5ac610b002fa67b4cced85644907ea6fe15195b3bda25" exitCode=0
Sep 19 08:23:46.977863 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.977851    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8aa0250-06dd-4d6f-b4dc-880851872780","Type":"ContainerDied","Data":"7affdec82c941242d6d5ac610b002fa67b4cced85644907ea6fe15195b3bda25"}
Sep 19 08:23:46.978351 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:46.977954    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8aa0250-06dd-4d6f-b4dc-880851872780","Type":"ContainerStarted","Data":"480542c85bf424a4353e7d90f0c5c5cc9bb95941c42c0fcb6c3c0105e6d1b2ac"}
Sep 19 08:23:47.936709 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:47.936631    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="670b77fa-df60-4f24-adbd-01ae35baf479" path="/var/lib/kubelet/pods/670b77fa-df60-4f24-adbd-01ae35baf479/volumes"
Sep 19 08:23:47.985633 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:47.985597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8aa0250-06dd-4d6f-b4dc-880851872780","Type":"ContainerStarted","Data":"a8a8445e85c3ab474639a59adafa2c6b2e62f9f058b75ac099b6d326328582d0"}
Sep 19 08:23:47.986043 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:47.985664    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8aa0250-06dd-4d6f-b4dc-880851872780","Type":"ContainerStarted","Data":"0df88e826cc39c2e9fbda961689a1d3863aef01b38bbaad20b1eeedbb2e9e1f2"}
Sep 19 08:23:47.986043 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:47.985691    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8aa0250-06dd-4d6f-b4dc-880851872780","Type":"ContainerStarted","Data":"6f1d51a2429ef6604f268605d7e320dd3ebe496fce6cfff1180b496da034e6bc"}
Sep 19 08:23:47.986043 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:47.985719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8aa0250-06dd-4d6f-b4dc-880851872780","Type":"ContainerStarted","Data":"e16a1bb3f1bbd163f75ba272ddc6824ce831a6433d75b5e33b7bc5922604c9d4"}
Sep 19 08:23:47.986043 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:47.985741    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8aa0250-06dd-4d6f-b4dc-880851872780","Type":"ContainerStarted","Data":"da24a7a88d312eb69d0268ad9fd3fe8d79b806645b784f34b414e5a52ba67066"}
Sep 19 08:23:47.986043 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:47.985763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d8aa0250-06dd-4d6f-b4dc-880851872780","Type":"ContainerStarted","Data":"0eba3db2ca0261d9fd953f66c822d0a487b86d7e167568bffea3a4929509b989"}
Sep 19 08:23:48.012170 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.012115    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.99631531 podStartE2EDuration="3.012097519s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.782209ms" isStatefulPod=false podCreationTimestamp="2026-09-19 08:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:23:48.010771977 +0000 UTC m=+180.421295014" watchObservedRunningTime="2026-09-19 08:23:48.012097519 +0000 UTC m=+180.422620560"
Sep 19 08:23:48.539784 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.539747    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"]
Sep 19 08:23:48.542898 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.542878    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.545229 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.545203    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 19 08:23:48.545483 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.545449    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 19 08:23:48.545540 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.545526    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 19 08:23:48.545596 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.545526    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2qszj\""
Sep 19 08:23:48.545669 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.545648    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 19 08:23:48.545897 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.545880    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 19 08:23:48.550319 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.550292    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 19 08:23:48.557071 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.557051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"]
Sep 19 08:23:48.705632 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.705589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-serving-certs-ca-bundle\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.705632 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.705634    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.705869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.705690    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.705869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.705744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-federate-client-tls\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.705869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.705780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8r4\" (UniqueName: \"kubernetes.io/projected/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-kube-api-access-gt8r4\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.705869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.705839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-telemeter-client-tls\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.706020 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.705880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-metrics-client-ca\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.706020 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.705918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-secret-telemeter-client\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.807390 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.807298    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-telemeter-client-tls\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.807390 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.807340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-metrics-client-ca\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.807390 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.807363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-secret-telemeter-client\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.807651 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.807418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-serving-certs-ca-bundle\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.807651 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.807442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.807651 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.807470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.807651 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.807588    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-federate-client-tls\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.807651 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.807621    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8r4\" (UniqueName: \"kubernetes.io/projected/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-kube-api-access-gt8r4\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.808716 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.808244    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-serving-certs-ca-bundle\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.814888 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.808655    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-metrics-client-ca\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.814888 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.809047    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.815789 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.815754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.815898 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.815765    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-federate-client-tls\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.815898 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.815804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-secret-telemeter-client\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.815898 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.815804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-telemeter-client-tls\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.817788 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.817765    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8r4\" (UniqueName: \"kubernetes.io/projected/ff541a40-d461-4c09-8d9e-a99bfa37a0a6-kube-api-access-gt8r4\") pod \"telemeter-client-56d65b9b7f-q9xtm\" (UID: \"ff541a40-d461-4c09-8d9e-a99bfa37a0a6\") " pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:48.852640 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:48.852610    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"
Sep 19 08:23:49.127668 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:49.127644    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm"]
Sep 19 08:23:49.130176 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:23:49.130145    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff541a40_d461_4c09_8d9e_a99bfa37a0a6.slice/crio-ddbecfb093e4f022436ab4b845d88641a90d2e2b89ba48400a4361148b6dbfac WatchSource:0}: Error finding container ddbecfb093e4f022436ab4b845d88641a90d2e2b89ba48400a4361148b6dbfac: Status 404 returned error can't find the container with id ddbecfb093e4f022436ab4b845d88641a90d2e2b89ba48400a4361148b6dbfac
Sep 19 08:23:49.996523 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:49.996490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm" event={"ID":"ff541a40-d461-4c09-8d9e-a99bfa37a0a6","Type":"ContainerStarted","Data":"ddbecfb093e4f022436ab4b845d88641a90d2e2b89ba48400a4361148b6dbfac"}
Sep 19 08:23:51.002939 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.002847    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm" event={"ID":"ff541a40-d461-4c09-8d9e-a99bfa37a0a6","Type":"ContainerStarted","Data":"558e6dec351c97b161da48e6f9364b3a19b6d730d03306e61fd1817f87f9da41"}
Sep 19 08:23:51.002939 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.002911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm" event={"ID":"ff541a40-d461-4c09-8d9e-a99bfa37a0a6","Type":"ContainerStarted","Data":"60785b8c8be7d70d85aa8e2dcf322dc64844bcca002d81fcfa65809638d57165"}
Sep 19 08:23:51.002939 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.002930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm" event={"ID":"ff541a40-d461-4c09-8d9e-a99bfa37a0a6","Type":"ContainerStarted","Data":"7a0efa0d3a0852cc6ed1baa669ba863c3c1f7975adb81f2369106c4ee3cfdb8b"}
Sep 19 08:23:51.023390 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.023331    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56d65b9b7f-q9xtm" podStartSLOduration=1.503596701 podStartE2EDuration="3.023310371s" totalImagesPullingTime="1.51971367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:23:51.02124889 +0000 UTC m=+183.431771890" watchObservedRunningTime="2026-09-19 08:23:51.023310371 +0000 UTC m=+183.433833376"
Sep 19 08:23:51.090642 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.090610    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c6f559995-7c6wj"]
Sep 19 08:23:51.092699 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.092679    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.101975 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.101955    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c6f559995-7c6wj"]
Sep 19 08:23:51.140548 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.140492    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:23:51.194383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.194333    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:23:51.231544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.231518    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6kvt\" (UniqueName: \"kubernetes.io/projected/046bb381-8f96-4105-918d-2437c101d056-kube-api-access-z6kvt\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.231705 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.231570    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-serving-cert\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.231705 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.231588    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-oauth-serving-cert\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.231705 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.231609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-oauth-config\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.231705 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.231625    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-service-ca\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.231705 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.231647    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-console-config\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.231705 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.231664    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-trusted-ca-bundle\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.333164 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.333125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-service-ca\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.333340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.333179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-console-config\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.333340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.333209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-trusted-ca-bundle\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.333340 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.333288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6kvt\" (UniqueName: \"kubernetes.io/projected/046bb381-8f96-4105-918d-2437c101d056-kube-api-access-z6kvt\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.333510 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.333395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-serving-cert\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.333510 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.333437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-oauth-serving-cert\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.333510 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.333475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-oauth-config\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.333953 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.333924    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-service-ca\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.334086 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.334044    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-console-config\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.334212 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.334192    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-oauth-serving-cert\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.334371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.334355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-trusted-ca-bundle\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.335919 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.335900    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-oauth-config\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.336017 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.335984    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-serving-cert\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.342726 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.342705    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6kvt\" (UniqueName: \"kubernetes.io/projected/046bb381-8f96-4105-918d-2437c101d056-kube-api-access-z6kvt\") pod \"console-6c6f559995-7c6wj\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") " pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.402758 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.402724    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:23:51.671485 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:51.671159    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c6f559995-7c6wj"]
Sep 19 08:23:51.673855 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:23:51.673818    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046bb381_8f96_4105_918d_2437c101d056.slice/crio-a672e054e20982505ec0a72173fbe662d05cd8f0a12189743396e88b6abbb719 WatchSource:0}: Error finding container a672e054e20982505ec0a72173fbe662d05cd8f0a12189743396e88b6abbb719: Status 404 returned error can't find the container with id a672e054e20982505ec0a72173fbe662d05cd8f0a12189743396e88b6abbb719
Sep 19 08:23:52.008647 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:52.008556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6f559995-7c6wj" event={"ID":"046bb381-8f96-4105-918d-2437c101d056","Type":"ContainerStarted","Data":"7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce"}
Sep 19 08:23:52.008647 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:52.008619    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6f559995-7c6wj" event={"ID":"046bb381-8f96-4105-918d-2437c101d056","Type":"ContainerStarted","Data":"a672e054e20982505ec0a72173fbe662d05cd8f0a12189743396e88b6abbb719"}
Sep 19 08:23:52.025185 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:23:52.025134    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c6f559995-7c6wj" podStartSLOduration=1.025114762 podStartE2EDuration="1.025114762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:23:52.023022053 +0000 UTC m=+184.433545052" watchObservedRunningTime="2026-09-19 08:23:52.025114762 +0000 UTC m=+184.435637762"
Sep 19 08:24:01.403116 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:01.403070    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:24:01.403532 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:01.403147    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:24:01.407955 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:01.407928    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:24:02.052103 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:02.052074    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:24:02.088807 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:02.088776    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54b64d5486-wlnnw"]
Sep 19 08:24:02.182893 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:02.182860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:24:02.235738 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:02.235705    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:24:02.291396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:02.291364    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:24:27.110024 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.109919    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54b64d5486-wlnnw" podUID="b5f0ad13-9117-4334-b675-3d7973f1c219" containerName="console" containerID="cri-o://1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1" gracePeriod=15
Sep 19 08:24:27.351875 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.351851    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:24:27.444310 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444190    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-console-config\") pod \"b5f0ad13-9117-4334-b675-3d7973f1c219\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") "
Sep 19 08:24:27.444483 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444321    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-service-ca\") pod \"b5f0ad13-9117-4334-b675-3d7973f1c219\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") "
Sep 19 08:24:27.444483 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84zp6\" (UniqueName: \"kubernetes.io/projected/b5f0ad13-9117-4334-b675-3d7973f1c219-kube-api-access-84zp6\") pod \"b5f0ad13-9117-4334-b675-3d7973f1c219\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") "
Sep 19 08:24:27.444483 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444413    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-oauth-serving-cert\") pod \"b5f0ad13-9117-4334-b675-3d7973f1c219\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") "
Sep 19 08:24:27.444483 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444445    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-oauth-config\") pod \"b5f0ad13-9117-4334-b675-3d7973f1c219\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") "
Sep 19 08:24:27.444483 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444470    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-trusted-ca-bundle\") pod \"b5f0ad13-9117-4334-b675-3d7973f1c219\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") "
Sep 19 08:24:27.444743 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444542    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-serving-cert\") pod \"b5f0ad13-9117-4334-b675-3d7973f1c219\" (UID: \"b5f0ad13-9117-4334-b675-3d7973f1c219\") "
Sep 19 08:24:27.444799 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444738    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-console-config" (OuterVolumeSpecName: "console-config") pod "b5f0ad13-9117-4334-b675-3d7973f1c219" (UID: "b5f0ad13-9117-4334-b675-3d7973f1c219"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:24:27.444799 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444757    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-service-ca" (OuterVolumeSpecName: "service-ca") pod "b5f0ad13-9117-4334-b675-3d7973f1c219" (UID: "b5f0ad13-9117-4334-b675-3d7973f1c219"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:24:27.444899 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444851    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b5f0ad13-9117-4334-b675-3d7973f1c219" (UID: "b5f0ad13-9117-4334-b675-3d7973f1c219"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:24:27.444942 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444921    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-console-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:24:27.444942 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.444941    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-service-ca\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:24:27.445142 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.445121    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b5f0ad13-9117-4334-b675-3d7973f1c219" (UID: "b5f0ad13-9117-4334-b675-3d7973f1c219"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:24:27.446799 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.446775    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b5f0ad13-9117-4334-b675-3d7973f1c219" (UID: "b5f0ad13-9117-4334-b675-3d7973f1c219"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:24:27.446896 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.446865    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b5f0ad13-9117-4334-b675-3d7973f1c219" (UID: "b5f0ad13-9117-4334-b675-3d7973f1c219"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:24:27.446896 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.446883    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f0ad13-9117-4334-b675-3d7973f1c219-kube-api-access-84zp6" (OuterVolumeSpecName: "kube-api-access-84zp6") pod "b5f0ad13-9117-4334-b675-3d7973f1c219" (UID: "b5f0ad13-9117-4334-b675-3d7973f1c219"). InnerVolumeSpecName "kube-api-access-84zp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:24:27.545887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.545835    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84zp6\" (UniqueName: \"kubernetes.io/projected/b5f0ad13-9117-4334-b675-3d7973f1c219-kube-api-access-84zp6\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:24:27.545887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.545879    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-oauth-serving-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:24:27.545887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.545889    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-oauth-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:24:27.545887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.545898    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5f0ad13-9117-4334-b675-3d7973f1c219-trusted-ca-bundle\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:24:27.545887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:27.545906    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5f0ad13-9117-4334-b675-3d7973f1c219-console-serving-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:24:28.161130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:28.161086    2559 generic.go:356] "Generic (PLEG): container finished" podID="b5f0ad13-9117-4334-b675-3d7973f1c219" containerID="1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1" exitCode=0
Sep 19 08:24:28.161560 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:28.161141    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54b64d5486-wlnnw"
Sep 19 08:24:28.161560 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:28.161163    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b64d5486-wlnnw" event={"ID":"b5f0ad13-9117-4334-b675-3d7973f1c219","Type":"ContainerDied","Data":"1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1"}
Sep 19 08:24:28.161560 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:28.161246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54b64d5486-wlnnw" event={"ID":"b5f0ad13-9117-4334-b675-3d7973f1c219","Type":"ContainerDied","Data":"48f376c6f46266e6a8e8fd19f666d8267ccdfcba61a08a02bfe03a9a92bdd199"}
Sep 19 08:24:28.161560 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:28.161284    2559 scope.go:117] "RemoveContainer" containerID="1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1"
Sep 19 08:24:28.170211 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:28.170190    2559 scope.go:117] "RemoveContainer" containerID="1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1"
Sep 19 08:24:28.170513 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:24:28.170490    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1\": container with ID starting with 1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1 not found: ID does not exist" containerID="1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1"
Sep 19 08:24:28.170588 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:28.170519    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1"} err="failed to get container status \"1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1\": rpc error: code = NotFound desc = could not find container \"1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1\": container with ID starting with 1561512ac48e587f83fe3b0f20f6ed9b9cefd6a880c4d0e0c79fa935e08f9ce1 not found: ID does not exist"
Sep 19 08:24:28.180113 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:28.180092    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54b64d5486-wlnnw"]
Sep 19 08:24:28.183626 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:28.183605    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54b64d5486-wlnnw"]
Sep 19 08:24:29.937326 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:29.937288    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f0ad13-9117-4334-b675-3d7973f1c219" path="/var/lib/kubelet/pods/b5f0ad13-9117-4334-b675-3d7973f1c219/volumes"
Sep 19 08:24:38.536729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:38.536696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:24:42.970771 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:42.970737    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:24:43.026725 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:43.026686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:24:58.707893 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:58.707857    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:24:58.774780 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:24:58.774752    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:00.579882 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.579849    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69d99d94d7-wjl8b"]
Sep 19 08:25:00.580394 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.580196    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f0ad13-9117-4334-b675-3d7973f1c219" containerName="console"
Sep 19 08:25:00.580394 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.580213    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f0ad13-9117-4334-b675-3d7973f1c219" containerName="console"
Sep 19 08:25:00.580394 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.580282    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f0ad13-9117-4334-b675-3d7973f1c219" containerName="console"
Sep 19 08:25:00.583443 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.583423    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.593995 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.593971    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d99d94d7-wjl8b"]
Sep 19 08:25:00.631380 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.631349    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:00.638573 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.638548    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-oauth-config\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.638694 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.638588    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-trusted-ca-bundle\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.638694 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.638625    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-oauth-serving-cert\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.638694 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.638668    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfpxz\" (UniqueName: \"kubernetes.io/projected/bdf9d149-78f2-491a-ba3b-a763f519fa28-kube-api-access-dfpxz\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.638813 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.638735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-service-ca\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.638813 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.638783    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-config\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.638881 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.638818    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-serving-cert\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.684633 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.684597    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:00.739761 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.739720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-serving-cert\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.739950 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.739779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-oauth-config\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.739950 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.739812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-trusted-ca-bundle\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.739950 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.739841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-oauth-serving-cert\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.739950 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.739882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfpxz\" (UniqueName: \"kubernetes.io/projected/bdf9d149-78f2-491a-ba3b-a763f519fa28-kube-api-access-dfpxz\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.739950 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.739933    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-service-ca\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.740198 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.739975    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-config\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.740774 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.740731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-oauth-serving-cert\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.740774 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.740731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-service-ca\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.740932 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.740809    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-config\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.740932 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.740830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-trusted-ca-bundle\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.742315 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.742296    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-serving-cert\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.742409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.742399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-oauth-config\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.748892 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.748861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfpxz\" (UniqueName: \"kubernetes.io/projected/bdf9d149-78f2-491a-ba3b-a763f519fa28-kube-api-access-dfpxz\") pod \"console-69d99d94d7-wjl8b\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") " pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:00.893405 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:00.893311    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:01.164434 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:01.164408    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69d99d94d7-wjl8b"]
Sep 19 08:25:01.167434 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:25:01.167401    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf9d149_78f2_491a_ba3b_a763f519fa28.slice/crio-f2a7b230f8094687b85268b16bdaad0576bd7cb25f1e45d141b70daf2e6cd82d WatchSource:0}: Error finding container f2a7b230f8094687b85268b16bdaad0576bd7cb25f1e45d141b70daf2e6cd82d: Status 404 returned error can't find the container with id f2a7b230f8094687b85268b16bdaad0576bd7cb25f1e45d141b70daf2e6cd82d
Sep 19 08:25:01.294822 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:01.294787    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d99d94d7-wjl8b" event={"ID":"bdf9d149-78f2-491a-ba3b-a763f519fa28","Type":"ContainerStarted","Data":"edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e"}
Sep 19 08:25:01.295018 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:01.294838    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d99d94d7-wjl8b" event={"ID":"bdf9d149-78f2-491a-ba3b-a763f519fa28","Type":"ContainerStarted","Data":"f2a7b230f8094687b85268b16bdaad0576bd7cb25f1e45d141b70daf2e6cd82d"}
Sep 19 08:25:01.315149 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:01.315091    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69d99d94d7-wjl8b" podStartSLOduration=1.315072893 podStartE2EDuration="1.315072893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:01.313861335 +0000 UTC m=+253.724384336" watchObservedRunningTime="2026-09-19 08:25:01.315072893 +0000 UTC m=+253.725595893"
Sep 19 08:25:10.667552 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:10.667514    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:10.738170 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:10.738139    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:10.893675 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:10.893631    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:10.893854 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:10.893719    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:10.898374 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:10.898352    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:11.338182 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:11.338155    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:25:11.392935 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:11.392894    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c6f559995-7c6wj"]
Sep 19 08:25:11.508167 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:11.508138    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:11.569147 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:11.569114    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:11.631088 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:11.631008    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:19.753687 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:19.753644    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:19.854744 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:19.854711    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:19.918970 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:19.918936    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:25:36.412586 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.412539    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c6f559995-7c6wj" podUID="046bb381-8f96-4105-918d-2437c101d056" containerName="console" containerID="cri-o://7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce" gracePeriod=15
Sep 19 08:25:36.661399 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.661374    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:25:36.753145 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753051    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-trusted-ca-bundle\") pod \"046bb381-8f96-4105-918d-2437c101d056\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") "
Sep 19 08:25:36.753145 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753094    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6kvt\" (UniqueName: \"kubernetes.io/projected/046bb381-8f96-4105-918d-2437c101d056-kube-api-access-z6kvt\") pod \"046bb381-8f96-4105-918d-2437c101d056\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") "
Sep 19 08:25:36.753145 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753128    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-oauth-serving-cert\") pod \"046bb381-8f96-4105-918d-2437c101d056\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") "
Sep 19 08:25:36.753470 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753153    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-service-ca\") pod \"046bb381-8f96-4105-918d-2437c101d056\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") "
Sep 19 08:25:36.753470 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753344    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-console-config\") pod \"046bb381-8f96-4105-918d-2437c101d056\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") "
Sep 19 08:25:36.753470 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753406    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-serving-cert\") pod \"046bb381-8f96-4105-918d-2437c101d056\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") "
Sep 19 08:25:36.753470 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753459    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-oauth-config\") pod \"046bb381-8f96-4105-918d-2437c101d056\" (UID: \"046bb381-8f96-4105-918d-2437c101d056\") "
Sep 19 08:25:36.753663 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753594    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "046bb381-8f96-4105-918d-2437c101d056" (UID: "046bb381-8f96-4105-918d-2437c101d056"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:36.753663 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753605    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-service-ca" (OuterVolumeSpecName: "service-ca") pod "046bb381-8f96-4105-918d-2437c101d056" (UID: "046bb381-8f96-4105-918d-2437c101d056"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:36.753663 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753621    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "046bb381-8f96-4105-918d-2437c101d056" (UID: "046bb381-8f96-4105-918d-2437c101d056"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:36.753810 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753714    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-console-config" (OuterVolumeSpecName: "console-config") pod "046bb381-8f96-4105-918d-2437c101d056" (UID: "046bb381-8f96-4105-918d-2437c101d056"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:36.753810 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753768    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-oauth-serving-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:25:36.753810 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753790    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-service-ca\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:25:36.753810 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753803    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-console-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:25:36.753954 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.753817    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/046bb381-8f96-4105-918d-2437c101d056-trusted-ca-bundle\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:25:36.755572 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.755548    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046bb381-8f96-4105-918d-2437c101d056-kube-api-access-z6kvt" (OuterVolumeSpecName: "kube-api-access-z6kvt") pod "046bb381-8f96-4105-918d-2437c101d056" (UID: "046bb381-8f96-4105-918d-2437c101d056"). InnerVolumeSpecName "kube-api-access-z6kvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:25:36.755672 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.755605    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "046bb381-8f96-4105-918d-2437c101d056" (UID: "046bb381-8f96-4105-918d-2437c101d056"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:25:36.755672 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.755618    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "046bb381-8f96-4105-918d-2437c101d056" (UID: "046bb381-8f96-4105-918d-2437c101d056"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:25:36.854923 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.854886    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-serving-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:25:36.854923 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.854916    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/046bb381-8f96-4105-918d-2437c101d056-console-oauth-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:25:36.854923 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:36.854925    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6kvt\" (UniqueName: \"kubernetes.io/projected/046bb381-8f96-4105-918d-2437c101d056-kube-api-access-z6kvt\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:25:37.437472 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:37.437441    2559 generic.go:356] "Generic (PLEG): container finished" podID="046bb381-8f96-4105-918d-2437c101d056" containerID="7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce" exitCode=0
Sep 19 08:25:37.437472 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:37.437479    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6f559995-7c6wj" event={"ID":"046bb381-8f96-4105-918d-2437c101d056","Type":"ContainerDied","Data":"7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce"}
Sep 19 08:25:37.437939 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:37.437515    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6f559995-7c6wj" event={"ID":"046bb381-8f96-4105-918d-2437c101d056","Type":"ContainerDied","Data":"a672e054e20982505ec0a72173fbe662d05cd8f0a12189743396e88b6abbb719"}
Sep 19 08:25:37.437939 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:37.437534    2559 scope.go:117] "RemoveContainer" containerID="7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce"
Sep 19 08:25:37.437939 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:37.437480    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6f559995-7c6wj"
Sep 19 08:25:37.446284 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:37.446242    2559 scope.go:117] "RemoveContainer" containerID="7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce"
Sep 19 08:25:37.446584 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:25:37.446563    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce\": container with ID starting with 7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce not found: ID does not exist" containerID="7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce"
Sep 19 08:25:37.446646 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:37.446593    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce"} err="failed to get container status \"7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce\": rpc error: code = NotFound desc = could not find container \"7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce\": container with ID starting with 7e7fb504fec34308846762cf272c4fecae4044e762733f92a28514bdea99ebce not found: ID does not exist"
Sep 19 08:25:37.459558 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:37.459534    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c6f559995-7c6wj"]
Sep 19 08:25:37.463302 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:37.463278    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c6f559995-7c6wj"]
Sep 19 08:25:37.937213 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:37.937178    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046bb381-8f96-4105-918d-2437c101d056" path="/var/lib/kubelet/pods/046bb381-8f96-4105-918d-2437c101d056/volumes"
Sep 19 08:25:49.808125 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:25:49.808098    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 19 08:26:21.858184 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:21.858149    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:22.953514 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:22.953480    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:44.110765 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:44.110688    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:44.224809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:44.224776    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:49.847196 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:49.847164    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:50.115632 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.115550    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:50.124782 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.124754    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-vsk9v"]
Sep 19 08:26:50.125111 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.125089    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046bb381-8f96-4105-918d-2437c101d056" containerName="console"
Sep 19 08:26:50.125111 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.125105    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="046bb381-8f96-4105-918d-2437c101d056" containerName="console"
Sep 19 08:26:50.125244 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.125187    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="046bb381-8f96-4105-918d-2437c101d056" containerName="console"
Sep 19 08:26:50.128062 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.128043    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:50.130876 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.130855    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 19 08:26:50.131090 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.131078    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 19 08:26:50.132163 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.132146    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-dq2rx\""
Sep 19 08:26:50.132280 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.132197    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 19 08:26:50.132280 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.132202    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 19 08:26:50.132453 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.132440    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 19 08:26:50.138401 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.138363    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-vsk9v"]
Sep 19 08:26:50.186663 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.186627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d0acee88-158e-4f45-9af9-225424f7a9da-cabundle0\") pod \"keda-operator-579c8958f7-vsk9v\" (UID: \"d0acee88-158e-4f45-9af9-225424f7a9da\") " pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:50.186862 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.186676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fbdt\" (UniqueName: \"kubernetes.io/projected/d0acee88-158e-4f45-9af9-225424f7a9da-kube-api-access-9fbdt\") pod \"keda-operator-579c8958f7-vsk9v\" (UID: \"d0acee88-158e-4f45-9af9-225424f7a9da\") " pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:50.186862 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.186748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d0acee88-158e-4f45-9af9-225424f7a9da-certificates\") pod \"keda-operator-579c8958f7-vsk9v\" (UID: \"d0acee88-158e-4f45-9af9-225424f7a9da\") " pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:50.215437 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.215404    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:50.287711 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.287665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d0acee88-158e-4f45-9af9-225424f7a9da-cabundle0\") pod \"keda-operator-579c8958f7-vsk9v\" (UID: \"d0acee88-158e-4f45-9af9-225424f7a9da\") " pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:50.287872 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.287752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fbdt\" (UniqueName: \"kubernetes.io/projected/d0acee88-158e-4f45-9af9-225424f7a9da-kube-api-access-9fbdt\") pod \"keda-operator-579c8958f7-vsk9v\" (UID: \"d0acee88-158e-4f45-9af9-225424f7a9da\") " pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:50.287872 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.287799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d0acee88-158e-4f45-9af9-225424f7a9da-certificates\") pod \"keda-operator-579c8958f7-vsk9v\" (UID: \"d0acee88-158e-4f45-9af9-225424f7a9da\") " pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:50.287953 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:26:50.287936    2559 secret.go:281] references non-existent secret key: ca.crt
Sep 19 08:26:50.287997 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:26:50.287957    2559 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 19 08:26:50.287997 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:26:50.287969    2559 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-vsk9v: references non-existent secret key: ca.crt
Sep 19 08:26:50.288090 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:26:50.288048    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0acee88-158e-4f45-9af9-225424f7a9da-certificates podName:d0acee88-158e-4f45-9af9-225424f7a9da nodeName:}" failed. No retries permitted until 2026-09-19 08:26:50.788025943 +0000 UTC m=+363.198548940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d0acee88-158e-4f45-9af9-225424f7a9da-certificates") pod "keda-operator-579c8958f7-vsk9v" (UID: "d0acee88-158e-4f45-9af9-225424f7a9da") : references non-existent secret key: ca.crt
Sep 19 08:26:50.288511 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.288489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d0acee88-158e-4f45-9af9-225424f7a9da-cabundle0\") pod \"keda-operator-579c8958f7-vsk9v\" (UID: \"d0acee88-158e-4f45-9af9-225424f7a9da\") " pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:50.298429 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.298398    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fbdt\" (UniqueName: \"kubernetes.io/projected/d0acee88-158e-4f45-9af9-225424f7a9da-kube-api-access-9fbdt\") pod \"keda-operator-579c8958f7-vsk9v\" (UID: \"d0acee88-158e-4f45-9af9-225424f7a9da\") " pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:50.325481 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.325453    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:50.724056 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.724024    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:50.793462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.793430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d0acee88-158e-4f45-9af9-225424f7a9da-certificates\") pod \"keda-operator-579c8958f7-vsk9v\" (UID: \"d0acee88-158e-4f45-9af9-225424f7a9da\") " pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:50.795914 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.795888    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d0acee88-158e-4f45-9af9-225424f7a9da-certificates\") pod \"keda-operator-579c8958f7-vsk9v\" (UID: \"d0acee88-158e-4f45-9af9-225424f7a9da\") " pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:50.891791 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.891759    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-lhcq8"]
Sep 19 08:26:50.895205 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.895183    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-lhcq8"
Sep 19 08:26:50.897474 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.897449    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 19 08:26:50.898723 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.898693    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:50.904812 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.904785    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-lhcq8"]
Sep 19 08:26:50.995241 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.995145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a93e309f-c67d-48d4-8b47-9120726465e6-certificates\") pod \"keda-admission-6c4d864b86-lhcq8\" (UID: \"a93e309f-c67d-48d4-8b47-9120726465e6\") " pod="openshift-keda/keda-admission-6c4d864b86-lhcq8"
Sep 19 08:26:50.995241 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:50.995207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52xmw\" (UniqueName: \"kubernetes.io/projected/a93e309f-c67d-48d4-8b47-9120726465e6-kube-api-access-52xmw\") pod \"keda-admission-6c4d864b86-lhcq8\" (UID: \"a93e309f-c67d-48d4-8b47-9120726465e6\") " pod="openshift-keda/keda-admission-6c4d864b86-lhcq8"
Sep 19 08:26:51.005803 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.005773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:51.038876 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.038850    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:51.096582 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.096202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a93e309f-c67d-48d4-8b47-9120726465e6-certificates\") pod \"keda-admission-6c4d864b86-lhcq8\" (UID: \"a93e309f-c67d-48d4-8b47-9120726465e6\") " pod="openshift-keda/keda-admission-6c4d864b86-lhcq8"
Sep 19 08:26:51.096582 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.096267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52xmw\" (UniqueName: \"kubernetes.io/projected/a93e309f-c67d-48d4-8b47-9120726465e6-kube-api-access-52xmw\") pod \"keda-admission-6c4d864b86-lhcq8\" (UID: \"a93e309f-c67d-48d4-8b47-9120726465e6\") " pod="openshift-keda/keda-admission-6c4d864b86-lhcq8"
Sep 19 08:26:51.096582 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:26:51.096456    2559 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 19 08:26:51.096582 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:26:51.096479    2559 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-lhcq8: secret "keda-admission-webhooks-certs" not found
Sep 19 08:26:51.096582 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:26:51.096558    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a93e309f-c67d-48d4-8b47-9120726465e6-certificates podName:a93e309f-c67d-48d4-8b47-9120726465e6 nodeName:}" failed. No retries permitted until 2026-09-19 08:26:51.596533801 +0000 UTC m=+364.007056786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a93e309f-c67d-48d4-8b47-9120726465e6-certificates") pod "keda-admission-6c4d864b86-lhcq8" (UID: "a93e309f-c67d-48d4-8b47-9120726465e6") : secret "keda-admission-webhooks-certs" not found
Sep 19 08:26:51.109085 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.109027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52xmw\" (UniqueName: \"kubernetes.io/projected/a93e309f-c67d-48d4-8b47-9120726465e6-kube-api-access-52xmw\") pod \"keda-admission-6c4d864b86-lhcq8\" (UID: \"a93e309f-c67d-48d4-8b47-9120726465e6\") " pod="openshift-keda/keda-admission-6c4d864b86-lhcq8"
Sep 19 08:26:51.309453 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.309347    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-vsk9v"]
Sep 19 08:26:51.312220 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:26:51.312191    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0acee88_158e_4f45_9af9_225424f7a9da.slice/crio-7398e3a191b40130e5ed1165e377eca8e0b731997400291dfc6c0e9511c92075 WatchSource:0}: Error finding container 7398e3a191b40130e5ed1165e377eca8e0b731997400291dfc6c0e9511c92075: Status 404 returned error can't find the container with id 7398e3a191b40130e5ed1165e377eca8e0b731997400291dfc6c0e9511c92075
Sep 19 08:26:51.313608 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.313589    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:51.313703 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.313648    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:26:51.430071 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.430037    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:51.541932 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.541904    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:51.601413 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.601330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a93e309f-c67d-48d4-8b47-9120726465e6-certificates\") pod \"keda-admission-6c4d864b86-lhcq8\" (UID: \"a93e309f-c67d-48d4-8b47-9120726465e6\") " pod="openshift-keda/keda-admission-6c4d864b86-lhcq8"
Sep 19 08:26:51.604021 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.604002    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a93e309f-c67d-48d4-8b47-9120726465e6-certificates\") pod \"keda-admission-6c4d864b86-lhcq8\" (UID: \"a93e309f-c67d-48d4-8b47-9120726465e6\") " pod="openshift-keda/keda-admission-6c4d864b86-lhcq8"
Sep 19 08:26:51.636687 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.636657    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:26:51.732567 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.732532    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-vsk9v" event={"ID":"d0acee88-158e-4f45-9af9-225424f7a9da","Type":"ContainerStarted","Data":"7398e3a191b40130e5ed1165e377eca8e0b731997400291dfc6c0e9511c92075"}
Sep 19 08:26:51.806432 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:51.806390    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-lhcq8"
Sep 19 08:26:52.095912 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:52.095825    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-lhcq8"]
Sep 19 08:26:52.100751 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:26:52.100130    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93e309f_c67d_48d4_8b47_9120726465e6.slice/crio-f5d22701e25a92f3499d441f44dabf50c330fa433e0999201082b938d2ee9acc WatchSource:0}: Error finding container f5d22701e25a92f3499d441f44dabf50c330fa433e0999201082b938d2ee9acc: Status 404 returned error can't find the container with id f5d22701e25a92f3499d441f44dabf50c330fa433e0999201082b938d2ee9acc
Sep 19 08:26:52.739280 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:52.739177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-lhcq8" event={"ID":"a93e309f-c67d-48d4-8b47-9120726465e6","Type":"ContainerStarted","Data":"f5d22701e25a92f3499d441f44dabf50c330fa433e0999201082b938d2ee9acc"}
Sep 19 08:26:55.753072 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:55.753024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-vsk9v" event={"ID":"d0acee88-158e-4f45-9af9-225424f7a9da","Type":"ContainerStarted","Data":"4ddf4e033fcdf06ee8231c1abe2471cd7934967d02059edbc195aa283326b3b7"}
Sep 19 08:26:55.753576 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:55.753132    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:26:55.754538 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:55.754510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-lhcq8" event={"ID":"a93e309f-c67d-48d4-8b47-9120726465e6","Type":"ContainerStarted","Data":"8c9c88b7f6481f2d5a9ec6f5d3184b1c9eaaaa13f9bcddf143fcc144511fdcf0"}
Sep 19 08:26:55.754662 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:55.754646    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-lhcq8"
Sep 19 08:26:55.768832 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:26:55.768785    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-vsk9v" podStartSLOduration=2.314451208 podStartE2EDuration="5.768769911s" totalImagesPullingTime="3.454318703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:26:55.768108398 +0000 UTC m=+368.178631399" watchObservedRunningTime="2026-09-19 08:26:55.768769911 +0000 UTC m=+368.179292915"
Sep 19 08:27:16.760359 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:16.760321    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-vsk9v"
Sep 19 08:27:16.760781 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:16.760439    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-lhcq8"
Sep 19 08:27:16.777706 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:16.777661    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-lhcq8" podStartSLOduration=24.070173364 podStartE2EDuration="26.777644674s" totalImagesPullingTime="2.70747131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:26:55.782868804 +0000 UTC m=+368.193391804" watchObservedRunningTime="2026-09-19 08:27:16.777644674 +0000 UTC m=+389.188167673"
Sep 19 08:27:18.388423 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.388382    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64b9cc5599-s69h8"]
Sep 19 08:27:18.450363 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.450328    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b9cc5599-s69h8"]
Sep 19 08:27:18.450544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.450455    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.450768 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.450749    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:18.550028 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.549991    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9xxq\" (UniqueName: \"kubernetes.io/projected/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-kube-api-access-h9xxq\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.550208 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.550047    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-trusted-ca-bundle\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.550208 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.550095    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-console-serving-cert\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.550208 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.550115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-service-ca\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.550208 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.550182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-console-oauth-config\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.550366 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.550215    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-console-config\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.550366 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.550278    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-oauth-serving-cert\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.589365 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.589330    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:18.651647 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.651573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-console-oauth-config\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.651647 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.651615    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-console-config\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.651857 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.651656    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-oauth-serving-cert\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.651857 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.651676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9xxq\" (UniqueName: \"kubernetes.io/projected/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-kube-api-access-h9xxq\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.651857 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.651707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-trusted-ca-bundle\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.651857 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.651735    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-console-serving-cert\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.651857 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.651756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-service-ca\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.652560 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.652502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-console-config\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.652716 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.652581    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-oauth-serving-cert\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.652716 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.652596    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-service-ca\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.652815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.652741    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-trusted-ca-bundle\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.653600 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.653576    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:18.654499 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.654474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-console-serving-cert\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.654567 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.654535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-console-oauth-config\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.660554 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.660532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9xxq\" (UniqueName: \"kubernetes.io/projected/d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9-kube-api-access-h9xxq\") pod \"console-64b9cc5599-s69h8\" (UID: \"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9\") " pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:18.759618 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:18.759586    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:19.031782 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:19.031755    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64b9cc5599-s69h8"]
Sep 19 08:27:19.034451 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:27:19.034426    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2794acc_d339_4f6c_a2d3_ff37cb3ca9a9.slice/crio-8d0f89ea1bfdb3e16841b63ceef38c6dbb2a3a9f2754094750c54b09d6150f56 WatchSource:0}: Error finding container 8d0f89ea1bfdb3e16841b63ceef38c6dbb2a3a9f2754094750c54b09d6150f56: Status 404 returned error can't find the container with id 8d0f89ea1bfdb3e16841b63ceef38c6dbb2a3a9f2754094750c54b09d6150f56
Sep 19 08:27:19.196362 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:19.196329    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:19.255752 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:19.255721    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:19.851659 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:19.851624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b9cc5599-s69h8" event={"ID":"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9","Type":"ContainerStarted","Data":"49217099def2e115359095f9b467fc558814ddd6cfd864b7b39302141e03715d"}
Sep 19 08:27:19.852047 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:19.851674    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64b9cc5599-s69h8" event={"ID":"d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9","Type":"ContainerStarted","Data":"8d0f89ea1bfdb3e16841b63ceef38c6dbb2a3a9f2754094750c54b09d6150f56"}
Sep 19 08:27:19.868697 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:19.868652    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64b9cc5599-s69h8" podStartSLOduration=1.868633037 podStartE2EDuration="1.868633037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:27:19.867959351 +0000 UTC m=+392.278482351" watchObservedRunningTime="2026-09-19 08:27:19.868633037 +0000 UTC m=+392.279156037"
Sep 19 08:27:28.760047 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:28.760006    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:28.760576 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:28.760071    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:28.765090 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:28.765064    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:28.889977 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:28.889948    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64b9cc5599-s69h8"
Sep 19 08:27:28.937503 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:28.937470    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d99d94d7-wjl8b"]
Sep 19 08:27:29.042850 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:29.042812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:29.161220 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:29.161183    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:29.261961 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:29.261910    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:38.739829 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:38.739787    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:38.796043 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:38.796011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:43.760487 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:43.760456    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:43.859177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:43.859142    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:27:53.961715 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:53.961651    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69d99d94d7-wjl8b" podUID="bdf9d149-78f2-491a-ba3b-a763f519fa28" containerName="console" containerID="cri-o://edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e" gracePeriod=15
Sep 19 08:27:54.211580 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.211556    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:27:54.272894 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.272815    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-oauth-config\") pod \"bdf9d149-78f2-491a-ba3b-a763f519fa28\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") "
Sep 19 08:27:54.272894 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.272852    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-oauth-serving-cert\") pod \"bdf9d149-78f2-491a-ba3b-a763f519fa28\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") "
Sep 19 08:27:54.273107 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.272911    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfpxz\" (UniqueName: \"kubernetes.io/projected/bdf9d149-78f2-491a-ba3b-a763f519fa28-kube-api-access-dfpxz\") pod \"bdf9d149-78f2-491a-ba3b-a763f519fa28\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") "
Sep 19 08:27:54.273107 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.272954    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-service-ca\") pod \"bdf9d149-78f2-491a-ba3b-a763f519fa28\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") "
Sep 19 08:27:54.273107 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.272993    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-trusted-ca-bundle\") pod \"bdf9d149-78f2-491a-ba3b-a763f519fa28\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") "
Sep 19 08:27:54.273107 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.273021    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-config\") pod \"bdf9d149-78f2-491a-ba3b-a763f519fa28\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") "
Sep 19 08:27:54.273107 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.273047    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-serving-cert\") pod \"bdf9d149-78f2-491a-ba3b-a763f519fa28\" (UID: \"bdf9d149-78f2-491a-ba3b-a763f519fa28\") "
Sep 19 08:27:54.273394 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.273365    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bdf9d149-78f2-491a-ba3b-a763f519fa28" (UID: "bdf9d149-78f2-491a-ba3b-a763f519fa28"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:54.273452 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.273429    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bdf9d149-78f2-491a-ba3b-a763f519fa28" (UID: "bdf9d149-78f2-491a-ba3b-a763f519fa28"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:54.273512 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.273438    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-service-ca" (OuterVolumeSpecName: "service-ca") pod "bdf9d149-78f2-491a-ba3b-a763f519fa28" (UID: "bdf9d149-78f2-491a-ba3b-a763f519fa28"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:54.273641 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.273621    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-config" (OuterVolumeSpecName: "console-config") pod "bdf9d149-78f2-491a-ba3b-a763f519fa28" (UID: "bdf9d149-78f2-491a-ba3b-a763f519fa28"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:54.275021 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.275003    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bdf9d149-78f2-491a-ba3b-a763f519fa28" (UID: "bdf9d149-78f2-491a-ba3b-a763f519fa28"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:54.275611 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.275583    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bdf9d149-78f2-491a-ba3b-a763f519fa28" (UID: "bdf9d149-78f2-491a-ba3b-a763f519fa28"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:54.275701 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.275609    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf9d149-78f2-491a-ba3b-a763f519fa28-kube-api-access-dfpxz" (OuterVolumeSpecName: "kube-api-access-dfpxz") pod "bdf9d149-78f2-491a-ba3b-a763f519fa28" (UID: "bdf9d149-78f2-491a-ba3b-a763f519fa28"). InnerVolumeSpecName "kube-api-access-dfpxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:27:54.374097 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.374058    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-trusted-ca-bundle\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:27:54.374097 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.374087    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:27:54.374097 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.374096    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-serving-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:27:54.374097 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.374105    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdf9d149-78f2-491a-ba3b-a763f519fa28-console-oauth-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:27:54.374406 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.374115    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-oauth-serving-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:27:54.374406 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.374123    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfpxz\" (UniqueName: \"kubernetes.io/projected/bdf9d149-78f2-491a-ba3b-a763f519fa28-kube-api-access-dfpxz\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:27:54.374406 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.374132    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf9d149-78f2-491a-ba3b-a763f519fa28-service-ca\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:27:54.991091 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.991052    2559 generic.go:356] "Generic (PLEG): container finished" podID="bdf9d149-78f2-491a-ba3b-a763f519fa28" containerID="edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e" exitCode=0
Sep 19 08:27:54.991525 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.991100    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69d99d94d7-wjl8b"
Sep 19 08:27:54.991525 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.991149    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d99d94d7-wjl8b" event={"ID":"bdf9d149-78f2-491a-ba3b-a763f519fa28","Type":"ContainerDied","Data":"edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e"}
Sep 19 08:27:54.991525 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.991202    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69d99d94d7-wjl8b" event={"ID":"bdf9d149-78f2-491a-ba3b-a763f519fa28","Type":"ContainerDied","Data":"f2a7b230f8094687b85268b16bdaad0576bd7cb25f1e45d141b70daf2e6cd82d"}
Sep 19 08:27:54.991525 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:54.991227    2559 scope.go:117] "RemoveContainer" containerID="edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e"
Sep 19 08:27:55.000525 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:55.000509    2559 scope.go:117] "RemoveContainer" containerID="edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e"
Sep 19 08:27:55.000794 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:27:55.000773    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e\": container with ID starting with edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e not found: ID does not exist" containerID="edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e"
Sep 19 08:27:55.000859 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:55.000801    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e"} err="failed to get container status \"edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e\": rpc error: code = NotFound desc = could not find container \"edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e\": container with ID starting with edd6d469c7ecb9fa0d6939211ef16a108281c9d168505b0fa5a41f737736869e not found: ID does not exist"
Sep 19 08:27:55.012944 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:55.012921    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69d99d94d7-wjl8b"]
Sep 19 08:27:55.015585 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:55.015565    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69d99d94d7-wjl8b"]
Sep 19 08:27:55.937914 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:27:55.937878    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdf9d149-78f2-491a-ba3b-a763f519fa28" path="/var/lib/kubelet/pods/bdf9d149-78f2-491a-ba3b-a763f519fa28/volumes"
Sep 19 08:28:05.279305 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:05.279200    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:05.380495 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:05.380458    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:10.303491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.303454    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:10.392789 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.392751    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:10.430665 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.430634    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-w2vfx"]
Sep 19 08:28:10.431149 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.431135    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdf9d149-78f2-491a-ba3b-a763f519fa28" containerName="console"
Sep 19 08:28:10.431194 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.431153    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdf9d149-78f2-491a-ba3b-a763f519fa28" containerName="console"
Sep 19 08:28:10.431272 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.431247    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdf9d149-78f2-491a-ba3b-a763f519fa28" containerName="console"
Sep 19 08:28:10.434467 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.434445    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-w2vfx"
Sep 19 08:28:10.437133 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.437112    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 19 08:28:10.437278 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.437231    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-btxrn\""
Sep 19 08:28:10.438138 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.438119    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 19 08:28:10.438220 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.438127    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 19 08:28:10.442916 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.442428    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-w2vfx"]
Sep 19 08:28:10.500722 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.500686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d36d14a0-5430-4555-ab10-4b151f2d7bc8-data\") pod \"seaweedfs-64568bcd49-w2vfx\" (UID: \"d36d14a0-5430-4555-ab10-4b151f2d7bc8\") " pod="kserve/seaweedfs-64568bcd49-w2vfx"
Sep 19 08:28:10.500898 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.500755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4zwp\" (UniqueName: \"kubernetes.io/projected/d36d14a0-5430-4555-ab10-4b151f2d7bc8-kube-api-access-g4zwp\") pod \"seaweedfs-64568bcd49-w2vfx\" (UID: \"d36d14a0-5430-4555-ab10-4b151f2d7bc8\") " pod="kserve/seaweedfs-64568bcd49-w2vfx"
Sep 19 08:28:10.507861 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.507836    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:10.601912 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.601828    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4zwp\" (UniqueName: \"kubernetes.io/projected/d36d14a0-5430-4555-ab10-4b151f2d7bc8-kube-api-access-g4zwp\") pod \"seaweedfs-64568bcd49-w2vfx\" (UID: \"d36d14a0-5430-4555-ab10-4b151f2d7bc8\") " pod="kserve/seaweedfs-64568bcd49-w2vfx"
Sep 19 08:28:10.602085 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.601918    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d36d14a0-5430-4555-ab10-4b151f2d7bc8-data\") pod \"seaweedfs-64568bcd49-w2vfx\" (UID: \"d36d14a0-5430-4555-ab10-4b151f2d7bc8\") " pod="kserve/seaweedfs-64568bcd49-w2vfx"
Sep 19 08:28:10.602351 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.602329    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d36d14a0-5430-4555-ab10-4b151f2d7bc8-data\") pod \"seaweedfs-64568bcd49-w2vfx\" (UID: \"d36d14a0-5430-4555-ab10-4b151f2d7bc8\") " pod="kserve/seaweedfs-64568bcd49-w2vfx"
Sep 19 08:28:10.611317 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.611282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:10.613126 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.613097    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4zwp\" (UniqueName: \"kubernetes.io/projected/d36d14a0-5430-4555-ab10-4b151f2d7bc8-kube-api-access-g4zwp\") pod \"seaweedfs-64568bcd49-w2vfx\" (UID: \"d36d14a0-5430-4555-ab10-4b151f2d7bc8\") " pod="kserve/seaweedfs-64568bcd49-w2vfx"
Sep 19 08:28:10.710597 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.710563    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:10.746346 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.746320    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-w2vfx"
Sep 19 08:28:10.936798 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:10.936722    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:11.018685 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:11.018541    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-w2vfx"]
Sep 19 08:28:11.021505 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:28:11.021472    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36d14a0_5430_4555_ab10_4b151f2d7bc8.slice/crio-1c2da2f326878e2d9b3b9c78b85147e7201b6aa2e0e10defc28861c4cb683d4c WatchSource:0}: Error finding container 1c2da2f326878e2d9b3b9c78b85147e7201b6aa2e0e10defc28861c4cb683d4c: Status 404 returned error can't find the container with id 1c2da2f326878e2d9b3b9c78b85147e7201b6aa2e0e10defc28861c4cb683d4c
Sep 19 08:28:11.021664 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:11.021641    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:11.058552 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:11.058517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-w2vfx" event={"ID":"d36d14a0-5430-4555-ab10-4b151f2d7bc8","Type":"ContainerStarted","Data":"1c2da2f326878e2d9b3b9c78b85147e7201b6aa2e0e10defc28861c4cb683d4c"}
Sep 19 08:28:11.083568 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:11.083542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:11.755632 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:11.755599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:28:14.073247 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:14.073207    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-w2vfx" event={"ID":"d36d14a0-5430-4555-ab10-4b151f2d7bc8","Type":"ContainerStarted","Data":"e0b534d7b138cc312b92ac75978c0fc0d447845863aa7b5e9da274ae77cfba99"}
Sep 19 08:28:14.073709 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:14.073310    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-w2vfx"
Sep 19 08:28:14.092636 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:14.092586    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-w2vfx" podStartSLOduration=1.380251968 podStartE2EDuration="4.092570355s" totalImagesPullingTime="2.712318387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:28:14.09057722 +0000 UTC m=+446.501100217" watchObservedRunningTime="2026-09-19 08:28:14.092570355 +0000 UTC m=+446.503093355"
Sep 19 08:28:20.080334 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:28:20.080302    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-w2vfx"
Sep 19 08:29:27.522403 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.522350    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:27.549273 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.549227    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-dqlqz"]
Sep 19 08:29:27.552797 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.552774    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dqlqz"
Sep 19 08:29:27.555906 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.555877    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 19 08:29:27.556181 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.556160    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-vq7n6\""
Sep 19 08:29:27.565428 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.565404    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dqlqz"]
Sep 19 08:29:27.568959 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.568779    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-b5vsn"]
Sep 19 08:29:27.572480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.572462    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:27.575606 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.575588    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 19 08:29:27.575812 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.575589    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 19 08:29:27.575883 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.575606    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-mmprb\""
Sep 19 08:29:27.581280 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.581201    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-b5vsn"]
Sep 19 08:29:27.606246 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.606202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:27.656357 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.656326    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzvh6\" (UniqueName: \"kubernetes.io/projected/5b1a44c2-0af7-4b1a-837c-00a9327ff5b3-kube-api-access-zzvh6\") pod \"odh-model-controller-7b966d7984-b5vsn\" (UID: \"5b1a44c2-0af7-4b1a-837c-00a9327ff5b3\") " pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:27.656512 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.656367    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1a44c2-0af7-4b1a-837c-00a9327ff5b3-metrics-certs\") pod \"odh-model-controller-7b966d7984-b5vsn\" (UID: \"5b1a44c2-0af7-4b1a-837c-00a9327ff5b3\") " pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:27.656512 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.656396    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008de914-d181-4d8c-83a2-dd742baa733b-tls-certs\") pod \"model-serving-api-7b954ffc95-dqlqz\" (UID: \"008de914-d181-4d8c-83a2-dd742baa733b\") " pod="kserve/model-serving-api-7b954ffc95-dqlqz"
Sep 19 08:29:27.656512 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.656448    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1a44c2-0af7-4b1a-837c-00a9327ff5b3-cert\") pod \"odh-model-controller-7b966d7984-b5vsn\" (UID: \"5b1a44c2-0af7-4b1a-837c-00a9327ff5b3\") " pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:27.656512 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.656493    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zm5\" (UniqueName: \"kubernetes.io/projected/008de914-d181-4d8c-83a2-dd742baa733b-kube-api-access-m9zm5\") pod \"model-serving-api-7b954ffc95-dqlqz\" (UID: \"008de914-d181-4d8c-83a2-dd742baa733b\") " pod="kserve/model-serving-api-7b954ffc95-dqlqz"
Sep 19 08:29:27.689681 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.689652    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:27.757641 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.757610    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzvh6\" (UniqueName: \"kubernetes.io/projected/5b1a44c2-0af7-4b1a-837c-00a9327ff5b3-kube-api-access-zzvh6\") pod \"odh-model-controller-7b966d7984-b5vsn\" (UID: \"5b1a44c2-0af7-4b1a-837c-00a9327ff5b3\") " pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:27.757837 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.757663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1a44c2-0af7-4b1a-837c-00a9327ff5b3-metrics-certs\") pod \"odh-model-controller-7b966d7984-b5vsn\" (UID: \"5b1a44c2-0af7-4b1a-837c-00a9327ff5b3\") " pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:27.757837 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.757707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008de914-d181-4d8c-83a2-dd742baa733b-tls-certs\") pod \"model-serving-api-7b954ffc95-dqlqz\" (UID: \"008de914-d181-4d8c-83a2-dd742baa733b\") " pod="kserve/model-serving-api-7b954ffc95-dqlqz"
Sep 19 08:29:27.757837 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:29:27.757817    2559 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 19 08:29:27.758002 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:29:27.757882    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/008de914-d181-4d8c-83a2-dd742baa733b-tls-certs podName:008de914-d181-4d8c-83a2-dd742baa733b nodeName:}" failed. No retries permitted until 2026-09-19 08:29:28.257858921 +0000 UTC m=+520.668381913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/008de914-d181-4d8c-83a2-dd742baa733b-tls-certs") pod "model-serving-api-7b954ffc95-dqlqz" (UID: "008de914-d181-4d8c-83a2-dd742baa733b") : secret "model-serving-api-tls" not found
Sep 19 08:29:27.758412 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.758375    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1a44c2-0af7-4b1a-837c-00a9327ff5b3-cert\") pod \"odh-model-controller-7b966d7984-b5vsn\" (UID: \"5b1a44c2-0af7-4b1a-837c-00a9327ff5b3\") " pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:27.758546 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.758445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zm5\" (UniqueName: \"kubernetes.io/projected/008de914-d181-4d8c-83a2-dd742baa733b-kube-api-access-m9zm5\") pod \"model-serving-api-7b954ffc95-dqlqz\" (UID: \"008de914-d181-4d8c-83a2-dd742baa733b\") " pod="kserve/model-serving-api-7b954ffc95-dqlqz"
Sep 19 08:29:27.762150 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.762119    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1a44c2-0af7-4b1a-837c-00a9327ff5b3-cert\") pod \"odh-model-controller-7b966d7984-b5vsn\" (UID: \"5b1a44c2-0af7-4b1a-837c-00a9327ff5b3\") " pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:27.764276 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.764230    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b1a44c2-0af7-4b1a-837c-00a9327ff5b3-metrics-certs\") pod \"odh-model-controller-7b966d7984-b5vsn\" (UID: \"5b1a44c2-0af7-4b1a-837c-00a9327ff5b3\") " pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:27.767985 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.767959    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zm5\" (UniqueName: \"kubernetes.io/projected/008de914-d181-4d8c-83a2-dd742baa733b-kube-api-access-m9zm5\") pod \"model-serving-api-7b954ffc95-dqlqz\" (UID: \"008de914-d181-4d8c-83a2-dd742baa733b\") " pod="kserve/model-serving-api-7b954ffc95-dqlqz"
Sep 19 08:29:27.768083 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.767986    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzvh6\" (UniqueName: \"kubernetes.io/projected/5b1a44c2-0af7-4b1a-837c-00a9327ff5b3-kube-api-access-zzvh6\") pod \"odh-model-controller-7b966d7984-b5vsn\" (UID: \"5b1a44c2-0af7-4b1a-837c-00a9327ff5b3\") " pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:27.790634 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.790571    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:27.886292 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:27.886233    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:28.166234 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:28.166208    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-b5vsn"]
Sep 19 08:29:28.168603 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:29:28.168574    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1a44c2_0af7_4b1a_837c_00a9327ff5b3.slice/crio-a7a51d8f62e3af316b935bae65de5b20e0a2b2ba3ca365a1c2f9221c97b24d92 WatchSource:0}: Error finding container a7a51d8f62e3af316b935bae65de5b20e0a2b2ba3ca365a1c2f9221c97b24d92: Status 404 returned error can't find the container with id a7a51d8f62e3af316b935bae65de5b20e0a2b2ba3ca365a1c2f9221c97b24d92
Sep 19 08:29:28.262693 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:28.262657    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008de914-d181-4d8c-83a2-dd742baa733b-tls-certs\") pod \"model-serving-api-7b954ffc95-dqlqz\" (UID: \"008de914-d181-4d8c-83a2-dd742baa733b\") " pod="kserve/model-serving-api-7b954ffc95-dqlqz"
Sep 19 08:29:28.265140 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:28.265116    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008de914-d181-4d8c-83a2-dd742baa733b-tls-certs\") pod \"model-serving-api-7b954ffc95-dqlqz\" (UID: \"008de914-d181-4d8c-83a2-dd742baa733b\") " pod="kserve/model-serving-api-7b954ffc95-dqlqz"
Sep 19 08:29:28.266229 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:28.266202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:28.377811 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:28.377775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-b5vsn" event={"ID":"5b1a44c2-0af7-4b1a-837c-00a9327ff5b3","Type":"ContainerStarted","Data":"a7a51d8f62e3af316b935bae65de5b20e0a2b2ba3ca365a1c2f9221c97b24d92"}
Sep 19 08:29:28.465888 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:28.465815    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dqlqz"
Sep 19 08:29:28.743246 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:28.743217    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dqlqz"]
Sep 19 08:29:28.745979 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:29:28.745945    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008de914_d181_4d8c_83a2_dd742baa733b.slice/crio-d1229336c6ea88c6b87ee2a4afa7aff3a83b2c05d70c210e15fbca2dcd0ded46 WatchSource:0}: Error finding container d1229336c6ea88c6b87ee2a4afa7aff3a83b2c05d70c210e15fbca2dcd0ded46: Status 404 returned error can't find the container with id d1229336c6ea88c6b87ee2a4afa7aff3a83b2c05d70c210e15fbca2dcd0ded46
Sep 19 08:29:29.243056 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:29.243016    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:29.304931 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:29.304893    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:29.386832 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:29.386710    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dqlqz" event={"ID":"008de914-d181-4d8c-83a2-dd742baa733b","Type":"ContainerStarted","Data":"d1229336c6ea88c6b87ee2a4afa7aff3a83b2c05d70c210e15fbca2dcd0ded46"}
Sep 19 08:29:31.753362 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:31.753331    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:29:32.403789 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:32.403747    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-b5vsn" event={"ID":"5b1a44c2-0af7-4b1a-837c-00a9327ff5b3","Type":"ContainerStarted","Data":"e84b5130d3f2dcf790150797b63671086ce900817986c158c101e69349e97f07"}
Sep 19 08:29:32.404000 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:32.403822    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:32.405271 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:32.405229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dqlqz" event={"ID":"008de914-d181-4d8c-83a2-dd742baa733b","Type":"ContainerStarted","Data":"53e9f57ab5e90a63265362cca3097ab6ee1a2401c5904dd8327c8ade762d1ddf"}
Sep 19 08:29:32.405380 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:32.405313    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-dqlqz"
Sep 19 08:29:32.425563 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:32.425512    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-b5vsn" podStartSLOduration=1.8768015249999999 podStartE2EDuration="5.425497029s" totalImagesPullingTime="3.548695504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:29:32.423791538 +0000 UTC m=+524.834314539" watchObservedRunningTime="2026-09-19 08:29:32.425497029 +0000 UTC m=+524.836020099"
Sep 19 08:29:32.444119 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:32.444075    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-dqlqz" podStartSLOduration=2.43599934 podStartE2EDuration="5.444057814s" totalImagesPullingTime="3.008058474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:29:32.442461814 +0000 UTC m=+524.852984817" watchObservedRunningTime="2026-09-19 08:29:32.444057814 +0000 UTC m=+524.854580811"
Sep 19 08:29:43.411211 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:43.411173    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-b5vsn"
Sep 19 08:29:43.413390 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:43.413368    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-dqlqz"
Sep 19 08:29:44.200683 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.200643    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-vtmbg"]
Sep 19 08:29:44.204036 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.204020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vtmbg"
Sep 19 08:29:44.206338 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.206315    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 19 08:29:44.212075 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.212051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vtmbg"]
Sep 19 08:29:44.309278 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.309218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cf2v\" (UniqueName: \"kubernetes.io/projected/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-kube-api-access-9cf2v\") pod \"s3-init-vtmbg\" (UID: \"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7\") " pod="kserve/s3-init-vtmbg"
Sep 19 08:29:44.309278 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.309287    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-model-cache\") pod \"s3-init-vtmbg\" (UID: \"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7\") " pod="kserve/s3-init-vtmbg"
Sep 19 08:29:44.410102 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.410065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cf2v\" (UniqueName: \"kubernetes.io/projected/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-kube-api-access-9cf2v\") pod \"s3-init-vtmbg\" (UID: \"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7\") " pod="kserve/s3-init-vtmbg"
Sep 19 08:29:44.410298 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.410127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-model-cache\") pod \"s3-init-vtmbg\" (UID: \"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7\") " pod="kserve/s3-init-vtmbg"
Sep 19 08:29:44.410530 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.410513    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-model-cache\") pod \"s3-init-vtmbg\" (UID: \"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7\") " pod="kserve/s3-init-vtmbg"
Sep 19 08:29:44.418589 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.418565    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cf2v\" (UniqueName: \"kubernetes.io/projected/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-kube-api-access-9cf2v\") pod \"s3-init-vtmbg\" (UID: \"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7\") " pod="kserve/s3-init-vtmbg"
Sep 19 08:29:44.524558 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.524478    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vtmbg"
Sep 19 08:29:44.796146 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:44.796120    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vtmbg"]
Sep 19 08:29:44.798268 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:29:44.798230    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podded8cbde_4d6c_4edb_a5bd_6df62838e3f7.slice/crio-ff5f657f3b4e240675bd5dd915660a5f670c4d58603b41a85bc140c4538078ef WatchSource:0}: Error finding container ff5f657f3b4e240675bd5dd915660a5f670c4d58603b41a85bc140c4538078ef: Status 404 returned error can't find the container with id ff5f657f3b4e240675bd5dd915660a5f670c4d58603b41a85bc140c4538078ef
Sep 19 08:29:45.461669 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:45.461632    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vtmbg" event={"ID":"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7","Type":"ContainerStarted","Data":"ff5f657f3b4e240675bd5dd915660a5f670c4d58603b41a85bc140c4538078ef"}
Sep 19 08:29:48.476990 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:48.476955    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vtmbg" event={"ID":"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7","Type":"ContainerStarted","Data":"20ac9e98674649eb1509ad08a03bd32d26619f4dbda9018129c5b79cf4bc4358"}
Sep 19 08:29:49.483164 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:49.483072    2559 generic.go:356] "Generic (PLEG): container finished" podID="ded8cbde-4d6c-4edb-a5bd-6df62838e3f7" containerID="20ac9e98674649eb1509ad08a03bd32d26619f4dbda9018129c5b79cf4bc4358" exitCode=0
Sep 19 08:29:49.483164 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:49.483137    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vtmbg" event={"ID":"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7","Type":"ContainerDied","Data":"20ac9e98674649eb1509ad08a03bd32d26619f4dbda9018129c5b79cf4bc4358"}
Sep 19 08:29:54.516187 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:54.516139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vtmbg" event={"ID":"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7","Type":"ContainerStarted","Data":"5d1698f6fe490ee6ade4d88a34f751097e21437672767c5d42041f1b8d9d3249"}
Sep 19 08:29:54.532878 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:29:54.532824    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-vtmbg" podStartSLOduration=1.769617168 podStartE2EDuration="10.532808441s" totalImagesPullingTime="7.351432953s" totalInitContainerRuntime="1.41175832s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:29:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:29:54.531675227 +0000 UTC m=+546.942198218" watchObservedRunningTime="2026-09-19 08:29:54.532808441 +0000 UTC m=+546.943331441"
Sep 19 08:30:59.816475 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:30:59.816439    2559 generic.go:356] "Generic (PLEG): container finished" podID="ded8cbde-4d6c-4edb-a5bd-6df62838e3f7" containerID="5d1698f6fe490ee6ade4d88a34f751097e21437672767c5d42041f1b8d9d3249" exitCode=0
Sep 19 08:30:59.816903 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:30:59.816515    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vtmbg" event={"ID":"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7","Type":"ContainerDied","Data":"5d1698f6fe490ee6ade4d88a34f751097e21437672767c5d42041f1b8d9d3249"}
Sep 19 08:31:00.945798 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:00.945774    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vtmbg"
Sep 19 08:31:00.992025 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:00.991944    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-model-cache\") pod \"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7\" (UID: \"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7\") "
Sep 19 08:31:00.992025 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:00.991995    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cf2v\" (UniqueName: \"kubernetes.io/projected/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-kube-api-access-9cf2v\") pod \"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7\" (UID: \"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7\") "
Sep 19 08:31:00.994228 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:00.994205    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-kube-api-access-9cf2v" (OuterVolumeSpecName: "kube-api-access-9cf2v") pod "ded8cbde-4d6c-4edb-a5bd-6df62838e3f7" (UID: "ded8cbde-4d6c-4edb-a5bd-6df62838e3f7"). InnerVolumeSpecName "kube-api-access-9cf2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:31:01.010866 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:01.010837    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "ded8cbde-4d6c-4edb-a5bd-6df62838e3f7" (UID: "ded8cbde-4d6c-4edb-a5bd-6df62838e3f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:31:01.093231 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:01.093150    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-model-cache\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:31:01.093231 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:01.093182    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cf2v\" (UniqueName: \"kubernetes.io/projected/ded8cbde-4d6c-4edb-a5bd-6df62838e3f7-kube-api-access-9cf2v\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:31:01.825869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:01.825838    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vtmbg"
Sep 19 08:31:01.826054 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:01.825837    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vtmbg" event={"ID":"ded8cbde-4d6c-4edb-a5bd-6df62838e3f7","Type":"ContainerDied","Data":"ff5f657f3b4e240675bd5dd915660a5f670c4d58603b41a85bc140c4538078ef"}
Sep 19 08:31:01.826054 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:01.825954    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff5f657f3b4e240675bd5dd915660a5f670c4d58603b41a85bc140c4538078ef"
Sep 19 08:31:02.263029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:02.262945    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:02.377828 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:02.377796    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:15.159090 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.159055    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:15.282000 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.281962    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"]
Sep 19 08:31:15.282424 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.282408    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded8cbde-4d6c-4edb-a5bd-6df62838e3f7" containerName="s3-init"
Sep 19 08:31:15.282424 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.282425    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded8cbde-4d6c-4edb-a5bd-6df62838e3f7" containerName="s3-init"
Sep 19 08:31:15.282542 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.282446    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded8cbde-4d6c-4edb-a5bd-6df62838e3f7" containerName="download-hf-model"
Sep 19 08:31:15.282542 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.282452    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded8cbde-4d6c-4edb-a5bd-6df62838e3f7" containerName="download-hf-model"
Sep 19 08:31:15.282634 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.282566    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded8cbde-4d6c-4edb-a5bd-6df62838e3f7" containerName="s3-init"
Sep 19 08:31:15.285132 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.285115    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.287812 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.287791    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:31:15.288042 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.288030    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:31:15.288122 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.288043    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:31:15.288122 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.288059    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:31:15.288246 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.288118    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-b4prd\""
Sep 19 08:31:15.288838 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.288820    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-48cee-predictor-serving-cert\""
Sep 19 08:31:15.288906 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.288820    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-48cee-kube-rbac-proxy-sar-config\""
Sep 19 08:31:15.300102 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.300075    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"]
Sep 19 08:31:15.352269 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.352030    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:15.390827 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.390789    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"]
Sep 19 08:31:15.394338 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.394318    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.396906 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.396881    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 19 08:31:15.397041 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.396926    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 19 08:31:15.405379 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.405355    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"]
Sep 19 08:31:15.429179 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.429103    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84c0373-130d-4373-ba33-6b30259fbd61-proxy-tls\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.429179 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.429145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.429179 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.429165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84c0373-130d-4373-ba33-6b30259fbd61-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.429448 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.429241    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.429448 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.429309    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vghp5\" (UniqueName: \"kubernetes.io/projected/f84c0373-130d-4373-ba33-6b30259fbd61-kube-api-access-vghp5\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.432758 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.432730    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"]
Sep 19 08:31:15.435285 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.435243    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.438302 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.438282    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 19 08:31:15.458848 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.458818    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"]
Sep 19 08:31:15.465347 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.465317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:15.530138 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530103    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695975f9-c937-45a6-9ddc-063a56c791d9-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.530138 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/695975f9-c937-45a6-9ddc-063a56c791d9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.530396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530170    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5r4h\" (UniqueName: \"kubernetes.io/projected/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-api-access-p5r4h\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.530396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84c0373-130d-4373-ba33-6b30259fbd61-proxy-tls\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.530396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpgl7\" (UniqueName: \"kubernetes.io/projected/695975f9-c937-45a6-9ddc-063a56c791d9-kube-api-access-vpgl7\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.530396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.530396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.530639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530429    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84c0373-130d-4373-ba33-6b30259fbd61-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.530639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d1d6908-0a2d-4bf1-a000-4113b683c111-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.530639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530501    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.530639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d1d6908-0a2d-4bf1-a000-4113b683c111-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.530639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530549    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vghp5\" (UniqueName: \"kubernetes.io/projected/f84c0373-130d-4373-ba33-6b30259fbd61-kube-api-access-vghp5\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.530639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.530978 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530697    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.530978 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530776    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.530978 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.530842    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84c0373-130d-4373-ba33-6b30259fbd61-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.531128 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.531108    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.531246 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.531226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.532852 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.532836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84c0373-130d-4373-ba33-6b30259fbd61-proxy-tls\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.544029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.543999    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vghp5\" (UniqueName: \"kubernetes.io/projected/f84c0373-130d-4373-ba33-6b30259fbd61-kube-api-access-vghp5\") pod \"isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.582156 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.582127    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:15.598169 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.598144    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:15.631490 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.631459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695975f9-c937-45a6-9ddc-063a56c791d9-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.631490 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.631496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/695975f9-c937-45a6-9ddc-063a56c791d9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.631739 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.631527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5r4h\" (UniqueName: \"kubernetes.io/projected/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-api-access-p5r4h\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.631739 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.631548    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpgl7\" (UniqueName: \"kubernetes.io/projected/695975f9-c937-45a6-9ddc-063a56c791d9-kube-api-access-vpgl7\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.631739 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.631576    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.631739 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.631601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d1d6908-0a2d-4bf1-a000-4113b683c111-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.631739 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.631642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d1d6908-0a2d-4bf1-a000-4113b683c111-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.631739 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.631664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.631739 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.631690    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.631739 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.631714    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.632134 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.631991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/695975f9-c937-45a6-9ddc-063a56c791d9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.632864 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.632375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.632864 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.632411    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d1d6908-0a2d-4bf1-a000-4113b683c111-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.632864 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.632752    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.632864 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.632809    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.633191 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.632878    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.633191 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:31:15.632904    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 19 08:31:15.633191 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:31:15.632982    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695975f9-c937-45a6-9ddc-063a56c791d9-proxy-tls podName:695975f9-c937-45a6-9ddc-063a56c791d9 nodeName:}" failed. No retries permitted until 2026-09-19 08:31:16.132963042 +0000 UTC m=+628.543486025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/695975f9-c937-45a6-9ddc-063a56c791d9-proxy-tls") pod "isvc-canary-rollback-predictor-7f7b644c9-58t2s" (UID: "695975f9-c937-45a6-9ddc-063a56c791d9") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 19 08:31:15.635051 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.635028    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d1d6908-0a2d-4bf1-a000-4113b683c111-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.640964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.640946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5r4h\" (UniqueName: \"kubernetes.io/projected/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-api-access-p5r4h\") pod \"isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.641138 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.641115    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpgl7\" (UniqueName: \"kubernetes.io/projected/695975f9-c937-45a6-9ddc-063a56c791d9-kube-api-access-vpgl7\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:15.683962 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.683889    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:15.707365 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.707336    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:15.882302 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.882271    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"]
Sep 19 08:31:15.883593 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:31:15.883561    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84c0373_130d_4373_ba33_6b30259fbd61.slice/crio-fae2fa9b133f05eafe29193724ba220f45ce04c849328ab3be82fa22b489970a WatchSource:0}: Error finding container fae2fa9b133f05eafe29193724ba220f45ce04c849328ab3be82fa22b489970a: Status 404 returned error can't find the container with id fae2fa9b133f05eafe29193724ba220f45ce04c849328ab3be82fa22b489970a
Sep 19 08:31:15.981401 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:15.981377    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"]
Sep 19 08:31:15.984411 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:31:15.984382    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1d6908_0a2d_4bf1_a000_4113b683c111.slice/crio-f51928a134039207556f53fe0b07719b15fa345b49948771530a23737dcc22a7 WatchSource:0}: Error finding container f51928a134039207556f53fe0b07719b15fa345b49948771530a23737dcc22a7: Status 404 returned error can't find the container with id f51928a134039207556f53fe0b07719b15fa345b49948771530a23737dcc22a7
Sep 19 08:31:16.001567 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:16.001545    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:16.137247 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:16.137197    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695975f9-c937-45a6-9ddc-063a56c791d9-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:16.139625 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:16.139600    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695975f9-c937-45a6-9ddc-063a56c791d9-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-58t2s\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:16.344920 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:16.344879    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:16.633551 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:16.633524    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"]
Sep 19 08:31:16.636172 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:31:16.636130    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695975f9_c937_45a6_9ddc_063a56c791d9.slice/crio-3cd149969f6319fbc1b5a468d1550199ea563cc928b724ef2352b5450757f6ae WatchSource:0}: Error finding container 3cd149969f6319fbc1b5a468d1550199ea563cc928b724ef2352b5450757f6ae: Status 404 returned error can't find the container with id 3cd149969f6319fbc1b5a468d1550199ea563cc928b724ef2352b5450757f6ae
Sep 19 08:31:16.896015 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:16.895916    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" event={"ID":"695975f9-c937-45a6-9ddc-063a56c791d9","Type":"ContainerStarted","Data":"f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0"}
Sep 19 08:31:16.896015 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:16.895985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" event={"ID":"695975f9-c937-45a6-9ddc-063a56c791d9","Type":"ContainerStarted","Data":"3cd149969f6319fbc1b5a468d1550199ea563cc928b724ef2352b5450757f6ae"}
Sep 19 08:31:16.897568 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:16.897538    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" event={"ID":"8d1d6908-0a2d-4bf1-a000-4113b683c111","Type":"ContainerStarted","Data":"585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d"}
Sep 19 08:31:16.897731 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:16.897596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" event={"ID":"8d1d6908-0a2d-4bf1-a000-4113b683c111","Type":"ContainerStarted","Data":"f51928a134039207556f53fe0b07719b15fa345b49948771530a23737dcc22a7"}
Sep 19 08:31:16.899553 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:16.899528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" event={"ID":"f84c0373-130d-4373-ba33-6b30259fbd61","Type":"ContainerStarted","Data":"df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945"}
Sep 19 08:31:16.899669 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:16.899580    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" event={"ID":"f84c0373-130d-4373-ba33-6b30259fbd61","Type":"ContainerStarted","Data":"fae2fa9b133f05eafe29193724ba220f45ce04c849328ab3be82fa22b489970a"}
Sep 19 08:31:19.917084 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:19.917050    2559 generic.go:356] "Generic (PLEG): container finished" podID="f84c0373-130d-4373-ba33-6b30259fbd61" containerID="df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945" exitCode=0
Sep 19 08:31:19.917497 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:19.917109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" event={"ID":"f84c0373-130d-4373-ba33-6b30259fbd61","Type":"ContainerDied","Data":"df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945"}
Sep 19 08:31:20.931477 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:20.931434    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerID="585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d" exitCode=0
Sep 19 08:31:20.931959 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:20.931631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" event={"ID":"8d1d6908-0a2d-4bf1-a000-4113b683c111","Type":"ContainerDied","Data":"585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d"}
Sep 19 08:31:20.934861 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:20.934832    2559 generic.go:356] "Generic (PLEG): container finished" podID="695975f9-c937-45a6-9ddc-063a56c791d9" containerID="f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0" exitCode=0
Sep 19 08:31:20.934940 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:20.934899    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" event={"ID":"695975f9-c937-45a6-9ddc-063a56c791d9","Type":"ContainerDied","Data":"f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0"}
Sep 19 08:31:41.144506 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:41.144465    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" event={"ID":"f84c0373-130d-4373-ba33-6b30259fbd61","Type":"ContainerStarted","Data":"cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903"}
Sep 19 08:31:41.146647 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:41.146614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" event={"ID":"695975f9-c937-45a6-9ddc-063a56c791d9","Type":"ContainerStarted","Data":"ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6"}
Sep 19 08:31:42.161065 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:42.161025    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" event={"ID":"8d1d6908-0a2d-4bf1-a000-4113b683c111","Type":"ContainerStarted","Data":"ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f"}
Sep 19 08:31:44.173780 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:44.173733    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" event={"ID":"695975f9-c937-45a6-9ddc-063a56c791d9","Type":"ContainerStarted","Data":"459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139"}
Sep 19 08:31:44.174276 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:44.173808    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:44.174276 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:44.173978    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:44.175605 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:44.175572    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:31:44.176239 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:44.176217    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" event={"ID":"8d1d6908-0a2d-4bf1-a000-4113b683c111","Type":"ContainerStarted","Data":"ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215"}
Sep 19 08:31:44.176362 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:44.176303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:44.176362 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:44.176340    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:44.177596 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:44.177551    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:31:44.178549 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:44.178531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" event={"ID":"f84c0373-130d-4373-ba33-6b30259fbd61","Type":"ContainerStarted","Data":"a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633"}
Sep 19 08:31:44.194791 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:44.194750    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podStartSLOduration=2.922501635 podStartE2EDuration="29.194736519s" totalImagesPullingTime="22.582795316s" totalInitContainerRuntime="3.689439568s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:31:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:31:44.192443746 +0000 UTC m=+656.602966748" watchObservedRunningTime="2026-09-19 08:31:44.194736519 +0000 UTC m=+656.605259518"
Sep 19 08:31:44.214793 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:44.214745    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podStartSLOduration=2.421024527 podStartE2EDuration="29.214733952s" totalImagesPullingTime="22.767890055s" totalInitContainerRuntime="4.02581937s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:31:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:31:44.213152277 +0000 UTC m=+656.623675271" watchObservedRunningTime="2026-09-19 08:31:44.214733952 +0000 UTC m=+656.625256951"
Sep 19 08:31:45.185479 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:45.185431    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:31:45.185958 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:45.185555    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:31:46.192563 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:46.192526    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" event={"ID":"f84c0373-130d-4373-ba33-6b30259fbd61","Type":"ContainerStarted","Data":"ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6"}
Sep 19 08:31:46.193178 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:46.192747    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:46.193178 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:46.192801    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:46.194280 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:46.194218    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:31:46.214132 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:46.214085    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podStartSLOduration=1.564379407 podStartE2EDuration="31.214066664s" totalImagesPullingTime="25.917612872s" totalInitContainerRuntime="3.732074385s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:31:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:31:46.212475369 +0000 UTC m=+658.622998381" watchObservedRunningTime="2026-09-19 08:31:46.214066664 +0000 UTC m=+658.624589663"
Sep 19 08:31:46.680491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:46.680457    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:47.197614 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:47.197580    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:47.198006 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:47.197764    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:31:47.199041 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:47.199006    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:31:48.203127 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:48.203076    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:31:48.203616 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:48.203531    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:31:48.207552 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:48.207530    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:31:49.208342 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:49.208294    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:31:49.208761 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:49.208575    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:31:50.190140 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:50.190108    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:31:50.190579 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:50.190560    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:31:50.190684 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:50.190627    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:31:50.191060 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:50.191040    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:31:52.168760 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:52.168701    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:58.554327 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:58.554294    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:31:59.208597 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:59.208543    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:31:59.209048 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:31:59.209019    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:32:00.191046 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:00.191011    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:32:00.191476 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:00.191009    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:32:09.209083 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:09.209039    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:32:09.209648 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:09.209621    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:32:10.190816 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:10.190778    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:32:10.191001 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:10.190930    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:32:19.208941 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:19.208888    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:32:19.209545 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:19.209380    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:32:20.191026 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:20.190987    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:32:20.191250 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:20.190988    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:32:29.209041 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:29.208996    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:32:29.209567 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:29.209513    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:32:30.191266 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:30.191214    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:32:30.191461 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:30.191214    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:32:39.208382 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:39.208278    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:32:39.208845 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:39.208821    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:32:40.191325 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:40.191281    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:32:40.191325 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:40.191306    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:32:49.208268 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:49.208218    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:32:49.208718 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:49.208639    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:32:50.191441 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:50.191405    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:32:50.191741 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:50.191410    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:32:59.208304 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:59.208227    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:32:59.208749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:59.208720    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:32:59.935492 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:32:59.935459    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 19 08:33:00.191201 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:00.191095    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:33:00.932549 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:00.932502    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:33:00.933026 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:00.932950    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:33:09.938140 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:09.938110    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:33:10.192110 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:10.192024    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:33:10.309583 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:10.309544    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"]
Sep 19 08:33:10.605454 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:10.605413    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kube-rbac-proxy" containerID="cri-o://459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139" gracePeriod=30
Sep 19 08:33:10.605673 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:10.605398    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container" containerID="cri-o://ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6" gracePeriod=30
Sep 19 08:33:10.933170 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:10.933085    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:33:10.933372 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:10.933353    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:33:11.612677 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:11.612644    2559 generic.go:356] "Generic (PLEG): container finished" podID="695975f9-c937-45a6-9ddc-063a56c791d9" containerID="459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139" exitCode=2
Sep 19 08:33:11.613063 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:11.612716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" event={"ID":"695975f9-c937-45a6-9ddc-063a56c791d9","Type":"ContainerDied","Data":"459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139"}
Sep 19 08:33:15.110662 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.110630    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"]
Sep 19 08:33:15.111072 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.110894    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" containerID="cri-o://ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f" gracePeriod=30
Sep 19 08:33:15.111072 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.110957    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kube-rbac-proxy" containerID="cri-o://ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215" gracePeriod=30
Sep 19 08:33:15.136662 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.136639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:33:15.177195 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.177159    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"]
Sep 19 08:33:15.181352 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.181330    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.185526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.185483    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 19 08:33:15.185670 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.185571    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 19 08:33:15.191969 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.191944    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"]
Sep 19 08:33:15.290920 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.290879    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgsf7\" (UniqueName: \"kubernetes.io/projected/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-api-access-tgsf7\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.291097 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.290954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.291097 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.290990    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb097685-bea2-4753-a435-c8fb5ca90d45-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.291097 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.291006    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.291218 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.291106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb097685-bea2-4753-a435-c8fb5ca90d45-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.392281 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.392175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb097685-bea2-4753-a435-c8fb5ca90d45-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.392281 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.392227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgsf7\" (UniqueName: \"kubernetes.io/projected/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-api-access-tgsf7\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.392475 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.392293    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.392475 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.392321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb097685-bea2-4753-a435-c8fb5ca90d45-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.392475 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.392348    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.392823 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.392789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb097685-bea2-4753-a435-c8fb5ca90d45-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.393069 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.393052    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.393112 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.393068    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.394811 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.394780    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb097685-bea2-4753-a435-c8fb5ca90d45-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.401501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.401481    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgsf7\" (UniqueName: \"kubernetes.io/projected/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-api-access-tgsf7\") pod \"isvc-canary-rollback-predictor-7f7b644c9-jlcv8\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.493052 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.493014    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:15.635327 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.635290    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerID="ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215" exitCode=2
Sep 19 08:33:15.635504 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.635350    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" event={"ID":"8d1d6908-0a2d-4bf1-a000-4113b683c111","Type":"ContainerDied","Data":"ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215"}
Sep 19 08:33:15.765684 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:15.765652    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"]
Sep 19 08:33:15.767847 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:33:15.767817    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb097685_bea2_4753_a435_c8fb5ca90d45.slice/crio-ebab42ad4d5206519d000b1e067a60273c87dbedd77e1e19363bcc3374fba1d1 WatchSource:0}: Error finding container ebab42ad4d5206519d000b1e067a60273c87dbedd77e1e19363bcc3374fba1d1: Status 404 returned error can't find the container with id ebab42ad4d5206519d000b1e067a60273c87dbedd77e1e19363bcc3374fba1d1
Sep 19 08:33:16.053010 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.052984    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:33:16.199907 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.199876    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/695975f9-c937-45a6-9ddc-063a56c791d9-kserve-provision-location\") pod \"695975f9-c937-45a6-9ddc-063a56c791d9\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") "
Sep 19 08:33:16.199907 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.199913    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpgl7\" (UniqueName: \"kubernetes.io/projected/695975f9-c937-45a6-9ddc-063a56c791d9-kube-api-access-vpgl7\") pod \"695975f9-c937-45a6-9ddc-063a56c791d9\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") "
Sep 19 08:33:16.200462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.200013    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-kube-rbac-proxy-sar-config\") pod \"695975f9-c937-45a6-9ddc-063a56c791d9\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") "
Sep 19 08:33:16.200462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.200055    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-cabundle-cert\") pod \"695975f9-c937-45a6-9ddc-063a56c791d9\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") "
Sep 19 08:33:16.200462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.200095    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695975f9-c937-45a6-9ddc-063a56c791d9-proxy-tls\") pod \"695975f9-c937-45a6-9ddc-063a56c791d9\" (UID: \"695975f9-c937-45a6-9ddc-063a56c791d9\") "
Sep 19 08:33:16.200462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.200186    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/695975f9-c937-45a6-9ddc-063a56c791d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "695975f9-c937-45a6-9ddc-063a56c791d9" (UID: "695975f9-c937-45a6-9ddc-063a56c791d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:33:16.200462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.200403    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "695975f9-c937-45a6-9ddc-063a56c791d9" (UID: "695975f9-c937-45a6-9ddc-063a56c791d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:33:16.200462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.200459    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/695975f9-c937-45a6-9ddc-063a56c791d9-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:16.200675 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.200506    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "695975f9-c937-45a6-9ddc-063a56c791d9" (UID: "695975f9-c937-45a6-9ddc-063a56c791d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:33:16.202396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.202361    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695975f9-c937-45a6-9ddc-063a56c791d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "695975f9-c937-45a6-9ddc-063a56c791d9" (UID: "695975f9-c937-45a6-9ddc-063a56c791d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:33:16.202526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.202427    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695975f9-c937-45a6-9ddc-063a56c791d9-kube-api-access-vpgl7" (OuterVolumeSpecName: "kube-api-access-vpgl7") pod "695975f9-c937-45a6-9ddc-063a56c791d9" (UID: "695975f9-c937-45a6-9ddc-063a56c791d9"). InnerVolumeSpecName "kube-api-access-vpgl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:33:16.301243 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.301166    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpgl7\" (UniqueName: \"kubernetes.io/projected/695975f9-c937-45a6-9ddc-063a56c791d9-kube-api-access-vpgl7\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:16.301243 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.301192    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:16.301243 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.301202    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/695975f9-c937-45a6-9ddc-063a56c791d9-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:16.301243 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.301213    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/695975f9-c937-45a6-9ddc-063a56c791d9-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:16.643110 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.643023    2559 generic.go:356] "Generic (PLEG): container finished" podID="695975f9-c937-45a6-9ddc-063a56c791d9" containerID="ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6" exitCode=0
Sep 19 08:33:16.643110 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.643089    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"
Sep 19 08:33:16.643368 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.643108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" event={"ID":"695975f9-c937-45a6-9ddc-063a56c791d9","Type":"ContainerDied","Data":"ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6"}
Sep 19 08:33:16.643368 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.643182    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s" event={"ID":"695975f9-c937-45a6-9ddc-063a56c791d9","Type":"ContainerDied","Data":"3cd149969f6319fbc1b5a468d1550199ea563cc928b724ef2352b5450757f6ae"}
Sep 19 08:33:16.643368 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.643219    2559 scope.go:117] "RemoveContainer" containerID="459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139"
Sep 19 08:33:16.645093 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.645063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" event={"ID":"cb097685-bea2-4753-a435-c8fb5ca90d45","Type":"ContainerStarted","Data":"89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75"}
Sep 19 08:33:16.645236 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.645124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" event={"ID":"cb097685-bea2-4753-a435-c8fb5ca90d45","Type":"ContainerStarted","Data":"ebab42ad4d5206519d000b1e067a60273c87dbedd77e1e19363bcc3374fba1d1"}
Sep 19 08:33:16.653174 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.653154    2559 scope.go:117] "RemoveContainer" containerID="ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6"
Sep 19 08:33:16.661008 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.660991    2559 scope.go:117] "RemoveContainer" containerID="f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0"
Sep 19 08:33:16.669984 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.669965    2559 scope.go:117] "RemoveContainer" containerID="459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139"
Sep 19 08:33:16.670297 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:16.670277    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139\": container with ID starting with 459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139 not found: ID does not exist" containerID="459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139"
Sep 19 08:33:16.670397 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.670309    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139"} err="failed to get container status \"459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139\": rpc error: code = NotFound desc = could not find container \"459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139\": container with ID starting with 459d80aee06a14c77c4eba43a7bd1c0c89c39c4b91ae7bfc0449d09bb541e139 not found: ID does not exist"
Sep 19 08:33:16.670397 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.670335    2559 scope.go:117] "RemoveContainer" containerID="ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6"
Sep 19 08:33:16.670642 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:16.670620    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6\": container with ID starting with ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6 not found: ID does not exist" containerID="ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6"
Sep 19 08:33:16.670688 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.670652    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6"} err="failed to get container status \"ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6\": rpc error: code = NotFound desc = could not find container \"ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6\": container with ID starting with ce8fded792b013169ffa64e7b8bd869eea40334f52bb1b8198a7d871d03c31a6 not found: ID does not exist"
Sep 19 08:33:16.670688 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.670673    2559 scope.go:117] "RemoveContainer" containerID="f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0"
Sep 19 08:33:16.670959 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:16.670937    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0\": container with ID starting with f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0 not found: ID does not exist" containerID="f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0"
Sep 19 08:33:16.671017 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.670966    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0"} err="failed to get container status \"f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0\": rpc error: code = NotFound desc = could not find container \"f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0\": container with ID starting with f2e9f57844721cfb1f59667c8f7bc6f58a363fb04e845bef148160d5aac002a0 not found: ID does not exist"
Sep 19 08:33:16.680329 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.680306    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"]
Sep 19 08:33:16.684544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:16.684524    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-58t2s"]
Sep 19 08:33:17.939287 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:17.939236    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" path="/var/lib/kubelet/pods/695975f9-c937-45a6-9ddc-063a56c791d9/volumes"
Sep 19 08:33:19.663619 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:19.663585    2559 generic.go:356] "Generic (PLEG): container finished" podID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerID="89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75" exitCode=0
Sep 19 08:33:19.663978 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:19.663646    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" event={"ID":"cb097685-bea2-4753-a435-c8fb5ca90d45","Type":"ContainerDied","Data":"89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75"}
Sep 19 08:33:20.186480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.186426    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 19 08:33:20.191844 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.191812    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 19 08:33:20.662832 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.662806    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:33:20.670732 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.670700    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerID="ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f" exitCode=0
Sep 19 08:33:20.671130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.670764    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"
Sep 19 08:33:20.671130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.670781    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" event={"ID":"8d1d6908-0a2d-4bf1-a000-4113b683c111","Type":"ContainerDied","Data":"ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f"}
Sep 19 08:33:20.671130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.670869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh" event={"ID":"8d1d6908-0a2d-4bf1-a000-4113b683c111","Type":"ContainerDied","Data":"f51928a134039207556f53fe0b07719b15fa345b49948771530a23737dcc22a7"}
Sep 19 08:33:20.671130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.670893    2559 scope.go:117] "RemoveContainer" containerID="ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215"
Sep 19 08:33:20.673305 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.673281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" event={"ID":"cb097685-bea2-4753-a435-c8fb5ca90d45","Type":"ContainerStarted","Data":"267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53"}
Sep 19 08:33:20.673400 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.673336    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" event={"ID":"cb097685-bea2-4753-a435-c8fb5ca90d45","Type":"ContainerStarted","Data":"fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519"}
Sep 19 08:33:20.673595 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.673577    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:20.673714 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.673627    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:20.674842 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.674817    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:33:20.679527 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.679510    2559 scope.go:117] "RemoveContainer" containerID="ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f"
Sep 19 08:33:20.688844 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.688816    2559 scope.go:117] "RemoveContainer" containerID="585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d"
Sep 19 08:33:20.696645 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.696617    2559 scope.go:117] "RemoveContainer" containerID="ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215"
Sep 19 08:33:20.696958 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:20.696939    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215\": container with ID starting with ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215 not found: ID does not exist" containerID="ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215"
Sep 19 08:33:20.697045 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.696964    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215"} err="failed to get container status \"ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215\": rpc error: code = NotFound desc = could not find container \"ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215\": container with ID starting with ff4d979351eef6527b3f2cde70f5cb3875e7c7015936d1cb62e1518f70d56215 not found: ID does not exist"
Sep 19 08:33:20.697045 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.696981    2559 scope.go:117] "RemoveContainer" containerID="ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f"
Sep 19 08:33:20.697232 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:20.697212    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f\": container with ID starting with ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f not found: ID does not exist" containerID="ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f"
Sep 19 08:33:20.697312 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.697234    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f"} err="failed to get container status \"ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f\": rpc error: code = NotFound desc = could not find container \"ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f\": container with ID starting with ed7da5165b4dd47417d3e0cb9cdd2ba97bdf2755acd43ef2ed1c98c8fcd2977f not found: ID does not exist"
Sep 19 08:33:20.697312 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.697271    2559 scope.go:117] "RemoveContainer" containerID="585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d"
Sep 19 08:33:20.697504 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:20.697487    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d\": container with ID starting with 585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d not found: ID does not exist" containerID="585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d"
Sep 19 08:33:20.697570 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.697507    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d"} err="failed to get container status \"585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d\": rpc error: code = NotFound desc = could not find container \"585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d\": container with ID starting with 585ff0364435ac31420336137d8ee94f03284013484299b2d74db6da2db3fd9d not found: ID does not exist"
Sep 19 08:33:20.701746 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.701666    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podStartSLOduration=1.939685018 podStartE2EDuration="5.701650283s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.761965265s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:33:20.699791595 +0000 UTC m=+753.110314596" watchObservedRunningTime="2026-09-19 08:33:20.701650283 +0000 UTC m=+753.112173282"
Sep 19 08:33:20.743896 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.743869    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-cabundle-cert\") pod \"8d1d6908-0a2d-4bf1-a000-4113b683c111\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") "
Sep 19 08:33:20.744037 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.743921    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d1d6908-0a2d-4bf1-a000-4113b683c111-kserve-provision-location\") pod \"8d1d6908-0a2d-4bf1-a000-4113b683c111\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") "
Sep 19 08:33:20.744037 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.743960    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d1d6908-0a2d-4bf1-a000-4113b683c111-proxy-tls\") pod \"8d1d6908-0a2d-4bf1-a000-4113b683c111\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") "
Sep 19 08:33:20.744037 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.744005    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-rbac-proxy-sar-config\") pod \"8d1d6908-0a2d-4bf1-a000-4113b683c111\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") "
Sep 19 08:33:20.744221 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.744037    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5r4h\" (UniqueName: \"kubernetes.io/projected/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-api-access-p5r4h\") pod \"8d1d6908-0a2d-4bf1-a000-4113b683c111\" (UID: \"8d1d6908-0a2d-4bf1-a000-4113b683c111\") "
Sep 19 08:33:20.744352 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.744326    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d1d6908-0a2d-4bf1-a000-4113b683c111" (UID: "8d1d6908-0a2d-4bf1-a000-4113b683c111"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:33:20.744418 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.744353    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1d6908-0a2d-4bf1-a000-4113b683c111-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d1d6908-0a2d-4bf1-a000-4113b683c111" (UID: "8d1d6908-0a2d-4bf1-a000-4113b683c111"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:33:20.744493 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.744469    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8d1d6908-0a2d-4bf1-a000-4113b683c111" (UID: "8d1d6908-0a2d-4bf1-a000-4113b683c111"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:33:20.744930 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.744734    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:20.744930 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.744759    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d1d6908-0a2d-4bf1-a000-4113b683c111-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:20.744930 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.744776    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d1d6908-0a2d-4bf1-a000-4113b683c111-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:20.746196 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.746170    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1d6908-0a2d-4bf1-a000-4113b683c111-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d1d6908-0a2d-4bf1-a000-4113b683c111" (UID: "8d1d6908-0a2d-4bf1-a000-4113b683c111"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:33:20.746435 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.746412    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-api-access-p5r4h" (OuterVolumeSpecName: "kube-api-access-p5r4h") pod "8d1d6908-0a2d-4bf1-a000-4113b683c111" (UID: "8d1d6908-0a2d-4bf1-a000-4113b683c111"). InnerVolumeSpecName "kube-api-access-p5r4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:33:20.846207 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.845531    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d1d6908-0a2d-4bf1-a000-4113b683c111-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:20.846207 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.845590    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5r4h\" (UniqueName: \"kubernetes.io/projected/8d1d6908-0a2d-4bf1-a000-4113b683c111-kube-api-access-p5r4h\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:20.860200 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.860168    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"]
Sep 19 08:33:20.860480 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.860447    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" containerID="cri-o://cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903" gracePeriod=30
Sep 19 08:33:20.860621 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.860478    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" containerID="cri-o://ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6" gracePeriod=30
Sep 19 08:33:20.860621 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.860500    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kube-rbac-proxy" containerID="cri-o://a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633" gracePeriod=30
Sep 19 08:33:20.881396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.881365    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:33:20.932679 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.932644    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:33:20.988732 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.988702    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:33:20.993840 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.993814    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"]
Sep 19 08:33:20.998307 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:20.998284    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-56575f6cc4-rpbnh"]
Sep 19 08:33:21.013399 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013373    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"]
Sep 19 08:33:21.013743 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013730    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container"
Sep 19 08:33:21.013802 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013744    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container"
Sep 19 08:33:21.013802 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013753    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kube-rbac-proxy"
Sep 19 08:33:21.013802 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013759    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kube-rbac-proxy"
Sep 19 08:33:21.013802 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013768    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="storage-initializer"
Sep 19 08:33:21.013802 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013774    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="storage-initializer"
Sep 19 08:33:21.013802 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013780    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container"
Sep 19 08:33:21.013802 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013785    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container"
Sep 19 08:33:21.013802 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013799    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kube-rbac-proxy"
Sep 19 08:33:21.013802 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013804    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kube-rbac-proxy"
Sep 19 08:33:21.014244 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013820    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="storage-initializer"
Sep 19 08:33:21.014244 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013826    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="storage-initializer"
Sep 19 08:33:21.014244 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013877    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kube-rbac-proxy"
Sep 19 08:33:21.014244 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013884    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" containerName="kserve-container"
Sep 19 08:33:21.014244 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013890    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kserve-container"
Sep 19 08:33:21.014244 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.013895    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="695975f9-c937-45a6-9ddc-063a56c791d9" containerName="kube-rbac-proxy"
Sep 19 08:33:21.017192 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.017176    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.019574 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.019554    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-619d8-predictor-serving-cert\""
Sep 19 08:33:21.019674 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.019558    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-619d8-kube-rbac-proxy-sar-config\""
Sep 19 08:33:21.027610 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.027588    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"]
Sep 19 08:33:21.048225 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.048192    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.048381 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.048238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.048381 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.048321    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtr6z\" (UniqueName: \"kubernetes.io/projected/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-api-access-dtr6z\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.048479 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.048390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173aa326-e2dc-467e-bc8a-84fa3385ee4a-proxy-tls\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.048479 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.048460    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.134415 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.134351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:33:21.149598 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.149566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtr6z\" (UniqueName: \"kubernetes.io/projected/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-api-access-dtr6z\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.149786 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.149640    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173aa326-e2dc-467e-bc8a-84fa3385ee4a-proxy-tls\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.149786 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.149699    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.149786 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.149766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.149953 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.149788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.150167 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:21.150140    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-serving-cert: secret "isvc-sklearn-graph-raw-619d8-predictor-serving-cert" not found
Sep 19 08:33:21.150276 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:21.150232    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/173aa326-e2dc-467e-bc8a-84fa3385ee4a-proxy-tls podName:173aa326-e2dc-467e-bc8a-84fa3385ee4a nodeName:}" failed. No retries permitted until 2026-09-19 08:33:21.650210211 +0000 UTC m=+754.060733203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/173aa326-e2dc-467e-bc8a-84fa3385ee4a-proxy-tls") pod "isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" (UID: "173aa326-e2dc-467e-bc8a-84fa3385ee4a") : secret "isvc-sklearn-graph-raw-619d8-predictor-serving-cert" not found
Sep 19 08:33:21.150450 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.150428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.150703 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.150676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.150841 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.150824    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.162441 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.162420    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtr6z\" (UniqueName: \"kubernetes.io/projected/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-api-access-dtr6z\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.512890 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.512848    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:33:21.655147 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.655114    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173aa326-e2dc-467e-bc8a-84fa3385ee4a-proxy-tls\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.657579 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.657546    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173aa326-e2dc-467e-bc8a-84fa3385ee4a-proxy-tls\") pod \"isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.670378 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.670344    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:33:21.680838 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.680812    2559 generic.go:356] "Generic (PLEG): container finished" podID="f84c0373-130d-4373-ba33-6b30259fbd61" containerID="a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633" exitCode=2
Sep 19 08:33:21.681179 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.680886    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" event={"ID":"f84c0373-130d-4373-ba33-6b30259fbd61","Type":"ContainerDied","Data":"a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633"}
Sep 19 08:33:21.682167 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.682141    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:33:21.928967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.928934    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:21.938270 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:21.938229    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d1d6908-0a2d-4bf1-a000-4113b683c111" path="/var/lib/kubelet/pods/8d1d6908-0a2d-4bf1-a000-4113b683c111/volumes"
Sep 19 08:33:22.209696 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:22.209671    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"]
Sep 19 08:33:22.214667 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:33:22.214513    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173aa326_e2dc_467e_bc8a_84fa3385ee4a.slice/crio-c4de1bea0ec710b26d01f36758e01aa21af88bdb5654516dc3c6d40554520d5a WatchSource:0}: Error finding container c4de1bea0ec710b26d01f36758e01aa21af88bdb5654516dc3c6d40554520d5a: Status 404 returned error can't find the container with id c4de1bea0ec710b26d01f36758e01aa21af88bdb5654516dc3c6d40554520d5a
Sep 19 08:33:22.688840 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:22.688803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" event={"ID":"173aa326-e2dc-467e-bc8a-84fa3385ee4a","Type":"ContainerStarted","Data":"0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c"}
Sep 19 08:33:22.689397 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:22.688861    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" event={"ID":"173aa326-e2dc-467e-bc8a-84fa3385ee4a","Type":"ContainerStarted","Data":"c4de1bea0ec710b26d01f36758e01aa21af88bdb5654516dc3c6d40554520d5a"}
Sep 19 08:33:23.203934 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:23.203891    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 19 08:33:26.686563 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:26.686480    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:33:26.687112 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:26.687082    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:33:26.710929 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:26.710899    2559 generic.go:356] "Generic (PLEG): container finished" podID="f84c0373-130d-4373-ba33-6b30259fbd61" containerID="cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903" exitCode=0
Sep 19 08:33:26.711105 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:26.710975    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" event={"ID":"f84c0373-130d-4373-ba33-6b30259fbd61","Type":"ContainerDied","Data":"cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903"}
Sep 19 08:33:26.712431 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:26.712409    2559 generic.go:356] "Generic (PLEG): container finished" podID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerID="0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c" exitCode=0
Sep 19 08:33:26.712554 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:26.712488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" event={"ID":"173aa326-e2dc-467e-bc8a-84fa3385ee4a","Type":"ContainerDied","Data":"0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c"}
Sep 19 08:33:27.720353 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:27.720314    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" event={"ID":"173aa326-e2dc-467e-bc8a-84fa3385ee4a","Type":"ContainerStarted","Data":"4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec"}
Sep 19 08:33:27.720759 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:27.720374    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" event={"ID":"173aa326-e2dc-467e-bc8a-84fa3385ee4a","Type":"ContainerStarted","Data":"e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b"}
Sep 19 08:33:27.720759 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:27.720654    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:27.720873 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:27.720803    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:27.722273 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:27.722235    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:33:27.765540 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:27.765486    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podStartSLOduration=3.942287472 podStartE2EDuration="7.76547063s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.823183158s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:33:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:33:27.762446893 +0000 UTC m=+760.172969890" watchObservedRunningTime="2026-09-19 08:33:27.76547063 +0000 UTC m=+760.175993630"
Sep 19 08:33:28.203212 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:28.203162    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 19 08:33:28.725716 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:28.725677    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:33:30.932525 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:30.932479    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:33:30.933004 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:30.932825    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:33:33.204086 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:33.204044    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 19 08:33:33.204576 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:33.204181    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:33:33.730857 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:33.730829    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:33:33.731496 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:33.731468    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:33:36.687505 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:36.687466    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:33:38.203663 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:38.203616    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 19 08:33:40.932765 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:40.932712    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:33:40.933181 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:40.933097    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:33:40.933222 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:40.933206    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:33:43.203400 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:43.203359    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 19 08:33:43.731852 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:43.731806    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:33:46.687773 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:46.687730    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:33:48.203730 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:48.203686    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 19 08:33:50.932587 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:50.932547    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent" probeResult="failure" output="Get \"http://10.133.0.32:9081/\": dial tcp 10.133.0.32:9081: connect: connection refused"
Sep 19 08:33:50.932945 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:50.932547    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 19 08:33:50.932945 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:50.932711    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:33:51.516555 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.516528    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:33:51.634134 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.634035    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84c0373-130d-4373-ba33-6b30259fbd61-proxy-tls\") pod \"f84c0373-130d-4373-ba33-6b30259fbd61\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") "
Sep 19 08:33:51.634134 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.634095    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-cabundle-cert\") pod \"f84c0373-130d-4373-ba33-6b30259fbd61\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") "
Sep 19 08:33:51.634134 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.634121    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vghp5\" (UniqueName: \"kubernetes.io/projected/f84c0373-130d-4373-ba33-6b30259fbd61-kube-api-access-vghp5\") pod \"f84c0373-130d-4373-ba33-6b30259fbd61\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") "
Sep 19 08:33:51.634540 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.634169    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-kube-rbac-proxy-sar-config\") pod \"f84c0373-130d-4373-ba33-6b30259fbd61\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") "
Sep 19 08:33:51.634540 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.634226    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84c0373-130d-4373-ba33-6b30259fbd61-kserve-provision-location\") pod \"f84c0373-130d-4373-ba33-6b30259fbd61\" (UID: \"f84c0373-130d-4373-ba33-6b30259fbd61\") "
Sep 19 08:33:51.634627 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.634578    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f84c0373-130d-4373-ba33-6b30259fbd61" (UID: "f84c0373-130d-4373-ba33-6b30259fbd61"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:33:51.634627 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.634596    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f84c0373-130d-4373-ba33-6b30259fbd61" (UID: "f84c0373-130d-4373-ba33-6b30259fbd61"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:33:51.634704 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.634600    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f84c0373-130d-4373-ba33-6b30259fbd61-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f84c0373-130d-4373-ba33-6b30259fbd61" (UID: "f84c0373-130d-4373-ba33-6b30259fbd61"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:33:51.636221 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.636200    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84c0373-130d-4373-ba33-6b30259fbd61-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f84c0373-130d-4373-ba33-6b30259fbd61" (UID: "f84c0373-130d-4373-ba33-6b30259fbd61"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:33:51.636333 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.636315    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84c0373-130d-4373-ba33-6b30259fbd61-kube-api-access-vghp5" (OuterVolumeSpecName: "kube-api-access-vghp5") pod "f84c0373-130d-4373-ba33-6b30259fbd61" (UID: "f84c0373-130d-4373-ba33-6b30259fbd61"). InnerVolumeSpecName "kube-api-access-vghp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:33:51.735709 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.735671    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f84c0373-130d-4373-ba33-6b30259fbd61-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:51.735709 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.735702    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:51.735709 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.735711    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vghp5\" (UniqueName: \"kubernetes.io/projected/f84c0373-130d-4373-ba33-6b30259fbd61-kube-api-access-vghp5\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:51.735956 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.735723    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f84c0373-130d-4373-ba33-6b30259fbd61-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:51.735956 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.735732    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f84c0373-130d-4373-ba33-6b30259fbd61-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:33:51.843015 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.842980    2559 generic.go:356] "Generic (PLEG): container finished" podID="f84c0373-130d-4373-ba33-6b30259fbd61" containerID="ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6" exitCode=0
Sep 19 08:33:51.843196 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.843043    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"
Sep 19 08:33:51.843196 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.843044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" event={"ID":"f84c0373-130d-4373-ba33-6b30259fbd61","Type":"ContainerDied","Data":"ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6"}
Sep 19 08:33:51.843196 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.843171    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt" event={"ID":"f84c0373-130d-4373-ba33-6b30259fbd61","Type":"ContainerDied","Data":"fae2fa9b133f05eafe29193724ba220f45ce04c849328ab3be82fa22b489970a"}
Sep 19 08:33:51.843323 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.843201    2559 scope.go:117] "RemoveContainer" containerID="ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6"
Sep 19 08:33:51.853702 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.853679    2559 scope.go:117] "RemoveContainer" containerID="a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633"
Sep 19 08:33:51.861839 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.861820    2559 scope.go:117] "RemoveContainer" containerID="cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903"
Sep 19 08:33:51.867955 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.867934    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"]
Sep 19 08:33:51.870631 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.870599    2559 scope.go:117] "RemoveContainer" containerID="df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945"
Sep 19 08:33:51.872390 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.872368    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-48cee-predictor-7cbd976774-t84jt"]
Sep 19 08:33:51.878333 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.878316    2559 scope.go:117] "RemoveContainer" containerID="ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6"
Sep 19 08:33:51.878651 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:51.878631    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6\": container with ID starting with ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6 not found: ID does not exist" containerID="ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6"
Sep 19 08:33:51.878725 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.878658    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6"} err="failed to get container status \"ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6\": rpc error: code = NotFound desc = could not find container \"ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6\": container with ID starting with ef8abaf6a50d1a813e7c094edc07374c1f70ee95881c15c3a75082be2462dae6 not found: ID does not exist"
Sep 19 08:33:51.878725 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.878676    2559 scope.go:117] "RemoveContainer" containerID="a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633"
Sep 19 08:33:51.878925 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:51.878908    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633\": container with ID starting with a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633 not found: ID does not exist" containerID="a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633"
Sep 19 08:33:51.878970 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.878931    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633"} err="failed to get container status \"a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633\": rpc error: code = NotFound desc = could not find container \"a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633\": container with ID starting with a957730a53fda156e1a8035e4595acc3ff23fc84a0985f55820ea26e4a613633 not found: ID does not exist"
Sep 19 08:33:51.878970 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.878947    2559 scope.go:117] "RemoveContainer" containerID="cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903"
Sep 19 08:33:51.879182 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:51.879163    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903\": container with ID starting with cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903 not found: ID does not exist" containerID="cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903"
Sep 19 08:33:51.879230 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.879188    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903"} err="failed to get container status \"cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903\": rpc error: code = NotFound desc = could not find container \"cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903\": container with ID starting with cf913a57d2c40a844c5479a854e569403761f8f7cb58a758d8d14b7ebc78c903 not found: ID does not exist"
Sep 19 08:33:51.879230 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.879203    2559 scope.go:117] "RemoveContainer" containerID="df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945"
Sep 19 08:33:51.879475 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:33:51.879460    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945\": container with ID starting with df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945 not found: ID does not exist" containerID="df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945"
Sep 19 08:33:51.879523 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.879481    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945"} err="failed to get container status \"df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945\": rpc error: code = NotFound desc = could not find container \"df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945\": container with ID starting with df8d05f9b60df4f3bd0017a9a8a36abad58e25156d4be95d5130919455fb0945 not found: ID does not exist"
Sep 19 08:33:51.938114 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:51.938025    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" path="/var/lib/kubelet/pods/f84c0373-130d-4373-ba33-6b30259fbd61/volumes"
Sep 19 08:33:53.732125 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:53.732088    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:33:56.687512 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:33:56.687469    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:34:03.731422 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:03.731381    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:34:06.687453 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:06.687412    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:34:13.732222 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:13.732184    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:34:16.688009 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:16.687956    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:34:23.732414 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:23.732371    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:34:26.687599 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:26.687562    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:34:33.731464 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:33.731425    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:34:36.687645 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:36.687604    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 19 08:34:43.732140 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:43.732097    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 19 08:34:44.933419 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:44.933392    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:34:45.217291 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:45.217195    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"]
Sep 19 08:34:45.242651 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:45.242612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:34:46.104584 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:46.104546    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container" containerID="cri-o://fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519" gracePeriod=30
Sep 19 08:34:46.105009 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:46.104551    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kube-rbac-proxy" containerID="cri-o://267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53" gracePeriod=30
Sep 19 08:34:46.683466 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:46.683429    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 19 08:34:46.933526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:46.933421    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:34:47.111003 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:47.110964    2559 generic.go:356] "Generic (PLEG): container finished" podID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerID="267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53" exitCode=2
Sep 19 08:34:47.111406 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:47.111035    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" event={"ID":"cb097685-bea2-4753-a435-c8fb5ca90d45","Type":"ContainerDied","Data":"267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53"}
Sep 19 08:34:50.370598 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.370559    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:34:50.407913 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.407882    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"]
Sep 19 08:34:50.408363 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.408317    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container"
Sep 19 08:34:50.408363 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.408362    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container"
Sep 19 08:34:50.408566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.408379    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="storage-initializer"
Sep 19 08:34:50.408566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.408387    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="storage-initializer"
Sep 19 08:34:50.408566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.408397    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kube-rbac-proxy"
Sep 19 08:34:50.408566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.408405    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kube-rbac-proxy"
Sep 19 08:34:50.408566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.408419    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent"
Sep 19 08:34:50.408566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.408427    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent"
Sep 19 08:34:50.408566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.408527    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kserve-container"
Sep 19 08:34:50.408566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.408539    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="agent"
Sep 19 08:34:50.408566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.408551    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f84c0373-130d-4373-ba33-6b30259fbd61" containerName="kube-rbac-proxy"
Sep 19 08:34:50.413099 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.413078    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.415597 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.415576    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 19 08:34:50.415712 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.415612    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 19 08:34:50.421614 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.421584    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"]
Sep 19 08:34:50.440674 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.440638    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:34:50.479514 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.479473    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.479729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.479540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/163f6283-d776-43d9-810b-e3443feccbbf-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.479729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.479637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/163f6283-d776-43d9-810b-e3443feccbbf-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.479729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.479692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49klm\" (UniqueName: \"kubernetes.io/projected/163f6283-d776-43d9-810b-e3443feccbbf-kube-api-access-49klm\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.479878 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.479738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.484184 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.484157    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"]
Sep 19 08:34:50.488554 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.488534    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.490824 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.490799    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 19 08:34:50.497061 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.497030    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"]
Sep 19 08:34:50.538874 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.538843    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:34:50.581346 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.581299    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.581552 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.581353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.581552 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.581427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.581552 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.581496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/163f6283-d776-43d9-810b-e3443feccbbf-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.581552 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.581527    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb10cf8e-9ab3-4539-878f-ef209aa23008-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.581790 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.581576    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.581790 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.581709    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/163f6283-d776-43d9-810b-e3443feccbbf-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.581790 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.581730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9zw7\" (UniqueName: \"kubernetes.io/projected/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-api-access-r9zw7\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.581790 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.581752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb10cf8e-9ab3-4539-878f-ef209aa23008-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.582009 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.581796    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49klm\" (UniqueName: \"kubernetes.io/projected/163f6283-d776-43d9-810b-e3443feccbbf-kube-api-access-49klm\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.582070 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.582057    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/163f6283-d776-43d9-810b-e3443feccbbf-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.582190 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.582169    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.582288 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.582171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.584105 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.584086    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/163f6283-d776-43d9-810b-e3443feccbbf-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.589129 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.589104    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49klm\" (UniqueName: \"kubernetes.io/projected/163f6283-d776-43d9-810b-e3443feccbbf-kube-api-access-49klm\") pod \"isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.683223 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.683117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb10cf8e-9ab3-4539-878f-ef209aa23008-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.683223 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.683156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.683223 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.683185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zw7\" (UniqueName: \"kubernetes.io/projected/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-api-access-r9zw7\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.683223 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.683201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb10cf8e-9ab3-4539-878f-ef209aa23008-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.683612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.683248    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.683612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.683579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb10cf8e-9ab3-4539-878f-ef209aa23008-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.683921 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.683903    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.684020 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.683911    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-cabundle-cert\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.686006 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.685982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb10cf8e-9ab3-4539-878f-ef209aa23008-proxy-tls\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.690924 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.690897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9zw7\" (UniqueName: \"kubernetes.io/projected/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-api-access-r9zw7\") pod \"isvc-canary-stop-predictor-5ffdccb94c-wl6vk\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.724795 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.724753    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:50.726342 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.726318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:34:50.800800 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.800764    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:50.999849 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:50.999823    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"]
Sep 19 08:34:51.001793 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:34:51.001764    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163f6283_d776_43d9_810b_e3443feccbbf.slice/crio-db50e8a41b226e8eaff24392d08c01602c5ac0072da7425521b3a0ac2f730432 WatchSource:0}: Error finding container db50e8a41b226e8eaff24392d08c01602c5ac0072da7425521b3a0ac2f730432: Status 404 returned error can't find the container with id db50e8a41b226e8eaff24392d08c01602c5ac0072da7425521b3a0ac2f730432
Sep 19 08:34:51.074738 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.074712    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"]
Sep 19 08:34:51.079232 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:34:51.079203    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb10cf8e_9ab3_4539_878f_ef209aa23008.slice/crio-f542146a9b0c222321f910ce157fa9dab0b61b9febfa7019fa845027042442ad WatchSource:0}: Error finding container f542146a9b0c222321f910ce157fa9dab0b61b9febfa7019fa845027042442ad: Status 404 returned error can't find the container with id f542146a9b0c222321f910ce157fa9dab0b61b9febfa7019fa845027042442ad
Sep 19 08:34:51.138086 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.138051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" event={"ID":"163f6283-d776-43d9-810b-e3443feccbbf","Type":"ContainerStarted","Data":"9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c"}
Sep 19 08:34:51.138393 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.138114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" event={"ID":"163f6283-d776-43d9-810b-e3443feccbbf","Type":"ContainerStarted","Data":"db50e8a41b226e8eaff24392d08c01602c5ac0072da7425521b3a0ac2f730432"}
Sep 19 08:34:51.139948 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.139926    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" event={"ID":"eb10cf8e-9ab3-4539-878f-ef209aa23008","Type":"ContainerStarted","Data":"f542146a9b0c222321f910ce157fa9dab0b61b9febfa7019fa845027042442ad"}
Sep 19 08:34:51.669759 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.669730    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:34:51.793824 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.793791    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb097685-bea2-4753-a435-c8fb5ca90d45-proxy-tls\") pod \"cb097685-bea2-4753-a435-c8fb5ca90d45\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") "
Sep 19 08:34:51.794023 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.793864    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb097685-bea2-4753-a435-c8fb5ca90d45-kserve-provision-location\") pod \"cb097685-bea2-4753-a435-c8fb5ca90d45\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") "
Sep 19 08:34:51.794023 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.793979    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-cabundle-cert\") pod \"cb097685-bea2-4753-a435-c8fb5ca90d45\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") "
Sep 19 08:34:51.794023 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.794014    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-rbac-proxy-sar-config\") pod \"cb097685-bea2-4753-a435-c8fb5ca90d45\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") "
Sep 19 08:34:51.794195 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.794074    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgsf7\" (UniqueName: \"kubernetes.io/projected/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-api-access-tgsf7\") pod \"cb097685-bea2-4753-a435-c8fb5ca90d45\" (UID: \"cb097685-bea2-4753-a435-c8fb5ca90d45\") "
Sep 19 08:34:51.794195 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.794153    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb097685-bea2-4753-a435-c8fb5ca90d45-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cb097685-bea2-4753-a435-c8fb5ca90d45" (UID: "cb097685-bea2-4753-a435-c8fb5ca90d45"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:34:51.794342 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.794321    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cb097685-bea2-4753-a435-c8fb5ca90d45" (UID: "cb097685-bea2-4753-a435-c8fb5ca90d45"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:34:51.794439 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.794426    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:34:51.794484 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.794442    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb097685-bea2-4753-a435-c8fb5ca90d45-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:34:51.794484 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.794449    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb097685-bea2-4753-a435-c8fb5ca90d45" (UID: "cb097685-bea2-4753-a435-c8fb5ca90d45"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:34:51.796097 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.796075    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb097685-bea2-4753-a435-c8fb5ca90d45-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb097685-bea2-4753-a435-c8fb5ca90d45" (UID: "cb097685-bea2-4753-a435-c8fb5ca90d45"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:34:51.796165 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.796133    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-api-access-tgsf7" (OuterVolumeSpecName: "kube-api-access-tgsf7") pod "cb097685-bea2-4753-a435-c8fb5ca90d45" (UID: "cb097685-bea2-4753-a435-c8fb5ca90d45"). InnerVolumeSpecName "kube-api-access-tgsf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:34:51.895233 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.895193    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb097685-bea2-4753-a435-c8fb5ca90d45-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:34:51.895233 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.895226    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:34:51.895233 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:51.895236    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgsf7\" (UniqueName: \"kubernetes.io/projected/cb097685-bea2-4753-a435-c8fb5ca90d45-kube-api-access-tgsf7\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:34:52.153865 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.153764    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" event={"ID":"eb10cf8e-9ab3-4539-878f-ef209aa23008","Type":"ContainerStarted","Data":"68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab"}
Sep 19 08:34:52.155834 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.155805    2559 generic.go:356] "Generic (PLEG): container finished" podID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerID="fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519" exitCode=0
Sep 19 08:34:52.155961 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.155869    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"
Sep 19 08:34:52.155961 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.155882    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" event={"ID":"cb097685-bea2-4753-a435-c8fb5ca90d45","Type":"ContainerDied","Data":"fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519"}
Sep 19 08:34:52.155961 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.155947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8" event={"ID":"cb097685-bea2-4753-a435-c8fb5ca90d45","Type":"ContainerDied","Data":"ebab42ad4d5206519d000b1e067a60273c87dbedd77e1e19363bcc3374fba1d1"}
Sep 19 08:34:52.156139 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.155985    2559 scope.go:117] "RemoveContainer" containerID="267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53"
Sep 19 08:34:52.164554 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.164537    2559 scope.go:117] "RemoveContainer" containerID="fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519"
Sep 19 08:34:52.172767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.172730    2559 scope.go:117] "RemoveContainer" containerID="89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75"
Sep 19 08:34:52.174801 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.174777    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"]
Sep 19 08:34:52.176550 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.176528    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f7b644c9-jlcv8"]
Sep 19 08:34:52.180713 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.180696    2559 scope.go:117] "RemoveContainer" containerID="267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53"
Sep 19 08:34:52.181000 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:34:52.180980    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53\": container with ID starting with 267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53 not found: ID does not exist" containerID="267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53"
Sep 19 08:34:52.181074 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.181014    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53"} err="failed to get container status \"267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53\": rpc error: code = NotFound desc = could not find container \"267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53\": container with ID starting with 267389afe597c5b4993a12bb76295fed5e9329a65f6cb90444d10607a906ef53 not found: ID does not exist"
Sep 19 08:34:52.181074 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.181039    2559 scope.go:117] "RemoveContainer" containerID="fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519"
Sep 19 08:34:52.181336 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:34:52.181314    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519\": container with ID starting with fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519 not found: ID does not exist" containerID="fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519"
Sep 19 08:34:52.181389 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.181349    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519"} err="failed to get container status \"fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519\": rpc error: code = NotFound desc = could not find container \"fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519\": container with ID starting with fdc31ceb83906e01c80ea4854de41750cd9c4267928013cfe25aa7395b409519 not found: ID does not exist"
Sep 19 08:34:52.181389 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.181371    2559 scope.go:117] "RemoveContainer" containerID="89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75"
Sep 19 08:34:52.181630 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:34:52.181613    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75\": container with ID starting with 89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75 not found: ID does not exist" containerID="89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75"
Sep 19 08:34:52.181674 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:52.181636    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75"} err="failed to get container status \"89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75\": rpc error: code = NotFound desc = could not find container \"89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75\": container with ID starting with 89c06035a6f8059725f45cd290af943efa293cc69592fb877e13eaef325d1d75 not found: ID does not exist"
Sep 19 08:34:53.938011 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:53.937973    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" path="/var/lib/kubelet/pods/cb097685-bea2-4753-a435-c8fb5ca90d45/volumes"
Sep 19 08:34:55.172387 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:55.172292    2559 generic.go:356] "Generic (PLEG): container finished" podID="163f6283-d776-43d9-810b-e3443feccbbf" containerID="9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c" exitCode=0
Sep 19 08:34:55.172387 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:55.172340    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" event={"ID":"163f6283-d776-43d9-810b-e3443feccbbf","Type":"ContainerDied","Data":"9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c"}
Sep 19 08:34:55.174175 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:55.174153    2559 generic.go:356] "Generic (PLEG): container finished" podID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerID="68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab" exitCode=0
Sep 19 08:34:55.174290 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:55.174227    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" event={"ID":"eb10cf8e-9ab3-4539-878f-ef209aa23008","Type":"ContainerDied","Data":"68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab"}
Sep 19 08:34:56.182068 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:56.182031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" event={"ID":"163f6283-d776-43d9-810b-e3443feccbbf","Type":"ContainerStarted","Data":"91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac"}
Sep 19 08:34:56.182570 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:56.182102    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" event={"ID":"163f6283-d776-43d9-810b-e3443feccbbf","Type":"ContainerStarted","Data":"908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc"}
Sep 19 08:34:56.182570 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:56.182425    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:56.182570 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:56.182550    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:34:56.184407 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:56.184377    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:34:56.184531 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:56.184504    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" event={"ID":"eb10cf8e-9ab3-4539-878f-ef209aa23008","Type":"ContainerStarted","Data":"1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e"}
Sep 19 08:34:56.184583 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:56.184565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" event={"ID":"eb10cf8e-9ab3-4539-878f-ef209aa23008","Type":"ContainerStarted","Data":"13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2"}
Sep 19 08:34:56.184824 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:56.184810    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:56.201051 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:56.201007    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podStartSLOduration=2.398399489 podStartE2EDuration="6.200992836s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.802593347s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:34:56.199719569 +0000 UTC m=+848.610242571" watchObservedRunningTime="2026-09-19 08:34:56.200992836 +0000 UTC m=+848.611515826"
Sep 19 08:34:56.217866 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:56.217818    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podStartSLOduration=2.378175018 podStartE2EDuration="6.217802444s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.839627426s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:34:56.215899631 +0000 UTC m=+848.626422632" watchObservedRunningTime="2026-09-19 08:34:56.217802444 +0000 UTC m=+848.628325444"
Sep 19 08:34:57.189912 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:57.189882    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:34:57.189912 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:57.189903    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:34:57.191354 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:57.191328    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:34:58.194739 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:34:58.194689    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:35:01.059727 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:01.059689    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:01.742709 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:01.742670    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:02.194672 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:02.194641    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:35:02.195311 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:02.195282    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:35:03.199355 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:03.199327    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:35:03.199983 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:03.199959    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:35:11.130391 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.130350    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:11.211248 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.211218    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"]
Sep 19 08:35:11.211505 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.211481    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container" containerID="cri-o://e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b" gracePeriod=30
Sep 19 08:35:11.211597 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.211542    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kube-rbac-proxy" containerID="cri-o://4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec" gracePeriod=30
Sep 19 08:35:11.234729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.234689    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:11.331966 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.331938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:11.335851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.335827    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"]
Sep 19 08:35:11.336209 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.336198    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kube-rbac-proxy"
Sep 19 08:35:11.336268 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.336210    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kube-rbac-proxy"
Sep 19 08:35:11.336268 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.336229    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="storage-initializer"
Sep 19 08:35:11.336268 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.336235    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="storage-initializer"
Sep 19 08:35:11.336268 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.336243    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container"
Sep 19 08:35:11.336268 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.336248    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container"
Sep 19 08:35:11.336425 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.336325    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kserve-container"
Sep 19 08:35:11.336425 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.336333    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb097685-bea2-4753-a435-c8fb5ca90d45" containerName="kube-rbac-proxy"
Sep 19 08:35:11.339440 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.339421    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.341679 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.341663    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-caf7a-predictor-serving-cert\""
Sep 19 08:35:11.341775 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.341724    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-caf7a-kube-rbac-proxy-sar-config\""
Sep 19 08:35:11.350873 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.350841    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"]
Sep 19 08:35:11.426463 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.426354    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:11.469809 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.469769    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.469995 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.469840    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.469995 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.469880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9d97776-bb73-4804-bb26-6566cef560b5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.469995 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.469946    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d97776-bb73-4804-bb26-6566cef560b5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.469995 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.469972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vd6c\" (UniqueName: \"kubernetes.io/projected/b9d97776-bb73-4804-bb26-6566cef560b5-kube-api-access-6vd6c\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.530640 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.530608    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:11.570871 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.570834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.571086 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.570888    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.571086 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.570917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9d97776-bb73-4804-bb26-6566cef560b5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.571086 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.570958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d97776-bb73-4804-bb26-6566cef560b5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.571086 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.570979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vd6c\" (UniqueName: \"kubernetes.io/projected/b9d97776-bb73-4804-bb26-6566cef560b5-kube-api-access-6vd6c\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.571521 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.571496    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9d97776-bb73-4804-bb26-6566cef560b5-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.571748 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.571727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.571807 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.571727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.573555 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.573535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d97776-bb73-4804-bb26-6566cef560b5-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.578875 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.578854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vd6c\" (UniqueName: \"kubernetes.io/projected/b9d97776-bb73-4804-bb26-6566cef560b5-kube-api-access-6vd6c\") pod \"isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.651682 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.651641    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:11.899177 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.899145    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:35:11.930942 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:11.930917    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"]
Sep 19 08:35:11.932528 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:35:11.932503    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d97776_bb73_4804_bb26_6566cef560b5.slice/crio-069a3a050340735cce415ce224b86dc4e844228b9666aeb216ded5bde29394ef WatchSource:0}: Error finding container 069a3a050340735cce415ce224b86dc4e844228b9666aeb216ded5bde29394ef: Status 404 returned error can't find the container with id 069a3a050340735cce415ce224b86dc4e844228b9666aeb216ded5bde29394ef
Sep 19 08:35:12.195545 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:12.195508    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:35:12.266542 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:12.266441    2559 generic.go:356] "Generic (PLEG): container finished" podID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerID="4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec" exitCode=2
Sep 19 08:35:12.266542 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:12.266511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" event={"ID":"173aa326-e2dc-467e-bc8a-84fa3385ee4a","Type":"ContainerDied","Data":"4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec"}
Sep 19 08:35:12.268228 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:12.268203    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" event={"ID":"b9d97776-bb73-4804-bb26-6566cef560b5","Type":"ContainerStarted","Data":"55c40e24a1e9904f1e9d1d0677a075d6a923e6b8eccba0d2c429c29e175bcafa"}
Sep 19 08:35:12.268380 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:12.268279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" event={"ID":"b9d97776-bb73-4804-bb26-6566cef560b5","Type":"ContainerStarted","Data":"069a3a050340735cce415ce224b86dc4e844228b9666aeb216ded5bde29394ef"}
Sep 19 08:35:13.200672 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:13.200631    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:35:13.725988 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:13.725942    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 19 08:35:16.289489 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.289455    2559 generic.go:356] "Generic (PLEG): container finished" podID="b9d97776-bb73-4804-bb26-6566cef560b5" containerID="55c40e24a1e9904f1e9d1d0677a075d6a923e6b8eccba0d2c429c29e175bcafa" exitCode=0
Sep 19 08:35:16.289897 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.289532    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" event={"ID":"b9d97776-bb73-4804-bb26-6566cef560b5","Type":"ContainerDied","Data":"55c40e24a1e9904f1e9d1d0677a075d6a923e6b8eccba0d2c429c29e175bcafa"}
Sep 19 08:35:16.673053 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.673029    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:35:16.818976 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.818940    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-rbac-proxy-sar-config\") pod \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") "
Sep 19 08:35:16.819151 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.819036    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kserve-provision-location\") pod \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") "
Sep 19 08:35:16.819151 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.819072    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtr6z\" (UniqueName: \"kubernetes.io/projected/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-api-access-dtr6z\") pod \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") "
Sep 19 08:35:16.819151 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.819126    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-cabundle-cert\") pod \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") "
Sep 19 08:35:16.819297 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.819150    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173aa326-e2dc-467e-bc8a-84fa3385ee4a-proxy-tls\") pod \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\" (UID: \"173aa326-e2dc-467e-bc8a-84fa3385ee4a\") "
Sep 19 08:35:16.819438 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.819410    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "173aa326-e2dc-467e-bc8a-84fa3385ee4a" (UID: "173aa326-e2dc-467e-bc8a-84fa3385ee4a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:35:16.819477 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.819450    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "173aa326-e2dc-467e-bc8a-84fa3385ee4a" (UID: "173aa326-e2dc-467e-bc8a-84fa3385ee4a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:35:16.819512 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.819499    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "173aa326-e2dc-467e-bc8a-84fa3385ee4a" (UID: "173aa326-e2dc-467e-bc8a-84fa3385ee4a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:35:16.819584 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.819562    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:35:16.819584 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.819579    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:35:16.821394 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.821368    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173aa326-e2dc-467e-bc8a-84fa3385ee4a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "173aa326-e2dc-467e-bc8a-84fa3385ee4a" (UID: "173aa326-e2dc-467e-bc8a-84fa3385ee4a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:35:16.821394 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.821388    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-api-access-dtr6z" (OuterVolumeSpecName: "kube-api-access-dtr6z") pod "173aa326-e2dc-467e-bc8a-84fa3385ee4a" (UID: "173aa326-e2dc-467e-bc8a-84fa3385ee4a"). InnerVolumeSpecName "kube-api-access-dtr6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:35:16.920800 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.920711    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtr6z\" (UniqueName: \"kubernetes.io/projected/173aa326-e2dc-467e-bc8a-84fa3385ee4a-kube-api-access-dtr6z\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:35:16.920800 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.920745    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/173aa326-e2dc-467e-bc8a-84fa3385ee4a-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:35:16.920800 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:16.920758    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173aa326-e2dc-467e-bc8a-84fa3385ee4a-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:35:17.297871 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.297830    2559 generic.go:356] "Generic (PLEG): container finished" podID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerID="e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b" exitCode=0
Sep 19 08:35:17.297871 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.297879    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"
Sep 19 08:35:17.298444 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.297914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" event={"ID":"173aa326-e2dc-467e-bc8a-84fa3385ee4a","Type":"ContainerDied","Data":"e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b"}
Sep 19 08:35:17.298444 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.297982    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj" event={"ID":"173aa326-e2dc-467e-bc8a-84fa3385ee4a","Type":"ContainerDied","Data":"c4de1bea0ec710b26d01f36758e01aa21af88bdb5654516dc3c6d40554520d5a"}
Sep 19 08:35:17.298444 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.298027    2559 scope.go:117] "RemoveContainer" containerID="4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec"
Sep 19 08:35:17.300498 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.300476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" event={"ID":"b9d97776-bb73-4804-bb26-6566cef560b5","Type":"ContainerStarted","Data":"ae912b1fb33be23317d6a082f9afc976cd59e139f6603bd111e6007481088022"}
Sep 19 08:35:17.300635 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.300534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" event={"ID":"b9d97776-bb73-4804-bb26-6566cef560b5","Type":"ContainerStarted","Data":"baa42acb89061e04f1362d865abf5bcd3c785ddabea114dd2a81c80642305dca"}
Sep 19 08:35:17.300852 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.300821    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:17.300926 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.300891    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:17.302150 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.302128    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 08:35:17.308308 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.308293    2559 scope.go:117] "RemoveContainer" containerID="e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b"
Sep 19 08:35:17.316089 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.316070    2559 scope.go:117] "RemoveContainer" containerID="0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c"
Sep 19 08:35:17.322649 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.322597    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podStartSLOduration=2.407378908 podStartE2EDuration="6.322578219s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.915199311s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:35:17.320781197 +0000 UTC m=+869.731304196" watchObservedRunningTime="2026-09-19 08:35:17.322578219 +0000 UTC m=+869.733101219"
Sep 19 08:35:17.325157 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.325139    2559 scope.go:117] "RemoveContainer" containerID="4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec"
Sep 19 08:35:17.325577 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:35:17.325554    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec\": container with ID starting with 4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec not found: ID does not exist" containerID="4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec"
Sep 19 08:35:17.325676 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.325582    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec"} err="failed to get container status \"4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec\": rpc error: code = NotFound desc = could not find container \"4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec\": container with ID starting with 4850cfb657bf1ff2e9a3cd67c486832ca154f87ab42bcda34da85ca3f7de9fec not found: ID does not exist"
Sep 19 08:35:17.325676 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.325599    2559 scope.go:117] "RemoveContainer" containerID="e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b"
Sep 19 08:35:17.325881 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:35:17.325860    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b\": container with ID starting with e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b not found: ID does not exist" containerID="e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b"
Sep 19 08:35:17.325946 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.325890    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b"} err="failed to get container status \"e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b\": rpc error: code = NotFound desc = could not find container \"e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b\": container with ID starting with e150bd78d2f23d81d42ee91c74790ccdf5cc3def73c00932eaabb0b1685ea15b not found: ID does not exist"
Sep 19 08:35:17.325946 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.325914    2559 scope.go:117] "RemoveContainer" containerID="0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c"
Sep 19 08:35:17.326167 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:35:17.326142    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c\": container with ID starting with 0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c not found: ID does not exist" containerID="0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c"
Sep 19 08:35:17.326277 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.326170    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c"} err="failed to get container status \"0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c\": rpc error: code = NotFound desc = could not find container \"0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c\": container with ID starting with 0b3fe3477afae2dbcacf44ad7e910b1cb8903009f5f624d46e6e2c5473e6423c not found: ID does not exist"
Sep 19 08:35:17.333864 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.333844    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"]
Sep 19 08:35:17.337442 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.337422    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-619d8-predictor-76d5bdbcbb-q2ltj"]
Sep 19 08:35:17.937919 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:17.937889    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" path="/var/lib/kubelet/pods/173aa326-e2dc-467e-bc8a-84fa3385ee4a/volumes"
Sep 19 08:35:18.306410 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:18.306369    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 08:35:22.195553 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:22.195507    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:35:23.200632 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:23.200592    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:35:23.310691 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:23.310659    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:35:23.311183 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:23.311155    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 08:35:32.195861 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:32.195774    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:35:33.200132 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:33.200090    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:35:33.311132 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:33.311086    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 08:35:42.195521 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:42.195478    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:35:43.200196 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:43.200155    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:35:43.311564 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:43.311521    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 08:35:52.195992 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:52.195952    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:35:53.200743 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:53.200703    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:35:53.311330 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:35:53.311285    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 08:36:02.195104 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:02.195063    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:36:03.200640 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:03.200598    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:36:03.311909 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:03.311868    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 08:36:12.195901 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:12.195856    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:36:13.200404 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:13.200364    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:36:13.311789 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:13.311749    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 08:36:13.932451 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:13.932398    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 19 08:36:18.933239 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:18.933205    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:36:23.311185 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:23.311141    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 08:36:23.937918 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:23.937887    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:36:24.063983 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:24.063949    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"]
Sep 19 08:36:24.064208 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:24.064174    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" containerID="cri-o://13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2" gracePeriod=30
Sep 19 08:36:24.064384 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:24.064223    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kube-rbac-proxy" containerID="cri-o://1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e" gracePeriod=30
Sep 19 08:36:24.634538 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:24.634502    2559 generic.go:356] "Generic (PLEG): container finished" podID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerID="1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e" exitCode=2
Sep 19 08:36:24.634927 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:24.634559    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" event={"ID":"eb10cf8e-9ab3-4539-878f-ef209aa23008","Type":"ContainerDied","Data":"1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e"}
Sep 19 08:36:25.421267 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:25.421218    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"]
Sep 19 08:36:25.421559 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:25.421523    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container" containerID="cri-o://908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc" gracePeriod=30
Sep 19 08:36:25.421683 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:25.421590    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kube-rbac-proxy" containerID="cri-o://91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac" gracePeriod=30
Sep 19 08:36:25.438733 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:25.438662    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:36:25.502592 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:25.502561    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:36:25.641770 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:25.641734    2559 generic.go:356] "Generic (PLEG): container finished" podID="163f6283-d776-43d9-810b-e3443feccbbf" containerID="91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac" exitCode=2
Sep 19 08:36:25.642164 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:25.641787    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" event={"ID":"163f6283-d776-43d9-810b-e3443feccbbf","Type":"ContainerDied","Data":"91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac"}
Sep 19 08:36:27.190277 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:27.190198    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 19 08:36:27.933398 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:27.933350    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 19 08:36:28.195972 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:28.195878    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 19 08:36:28.932883 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:28.932843    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 19 08:36:29.410394 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.410361    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:36:29.485562 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.485530    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb10cf8e-9ab3-4539-878f-ef209aa23008-kserve-provision-location\") pod \"eb10cf8e-9ab3-4539-878f-ef209aa23008\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") "
Sep 19 08:36:29.485760 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.485573    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9zw7\" (UniqueName: \"kubernetes.io/projected/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-api-access-r9zw7\") pod \"eb10cf8e-9ab3-4539-878f-ef209aa23008\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") "
Sep 19 08:36:29.485760 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.485643    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-rbac-proxy-sar-config\") pod \"eb10cf8e-9ab3-4539-878f-ef209aa23008\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") "
Sep 19 08:36:29.485760 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.485672    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-cabundle-cert\") pod \"eb10cf8e-9ab3-4539-878f-ef209aa23008\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") "
Sep 19 08:36:29.485760 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.485697    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb10cf8e-9ab3-4539-878f-ef209aa23008-proxy-tls\") pod \"eb10cf8e-9ab3-4539-878f-ef209aa23008\" (UID: \"eb10cf8e-9ab3-4539-878f-ef209aa23008\") "
Sep 19 08:36:29.485994 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.485946    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb10cf8e-9ab3-4539-878f-ef209aa23008-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb10cf8e-9ab3-4539-878f-ef209aa23008" (UID: "eb10cf8e-9ab3-4539-878f-ef209aa23008"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:29.486069 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.486046    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb10cf8e-9ab3-4539-878f-ef209aa23008" (UID: "eb10cf8e-9ab3-4539-878f-ef209aa23008"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:36:29.486108 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.486043    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb10cf8e-9ab3-4539-878f-ef209aa23008" (UID: "eb10cf8e-9ab3-4539-878f-ef209aa23008"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:36:29.487727 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.487703    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb10cf8e-9ab3-4539-878f-ef209aa23008-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb10cf8e-9ab3-4539-878f-ef209aa23008" (UID: "eb10cf8e-9ab3-4539-878f-ef209aa23008"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:36:29.487828 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.487766    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-api-access-r9zw7" (OuterVolumeSpecName: "kube-api-access-r9zw7") pod "eb10cf8e-9ab3-4539-878f-ef209aa23008" (UID: "eb10cf8e-9ab3-4539-878f-ef209aa23008"). InnerVolumeSpecName "kube-api-access-r9zw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:36:29.587049 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.587008    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:36:29.587049 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.587041    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb10cf8e-9ab3-4539-878f-ef209aa23008-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:36:29.587049 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.587051    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb10cf8e-9ab3-4539-878f-ef209aa23008-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:36:29.587049 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.587060    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb10cf8e-9ab3-4539-878f-ef209aa23008-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:36:29.587359 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.587069    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9zw7\" (UniqueName: \"kubernetes.io/projected/eb10cf8e-9ab3-4539-878f-ef209aa23008-kube-api-access-r9zw7\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:36:29.665408 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.665371    2559 generic.go:356] "Generic (PLEG): container finished" podID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerID="13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2" exitCode=0
Sep 19 08:36:29.665602 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.665432    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"
Sep 19 08:36:29.665602 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.665433    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" event={"ID":"eb10cf8e-9ab3-4539-878f-ef209aa23008","Type":"ContainerDied","Data":"13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2"}
Sep 19 08:36:29.665602 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.665568    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk" event={"ID":"eb10cf8e-9ab3-4539-878f-ef209aa23008","Type":"ContainerDied","Data":"f542146a9b0c222321f910ce157fa9dab0b61b9febfa7019fa845027042442ad"}
Sep 19 08:36:29.665777 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.665606    2559 scope.go:117] "RemoveContainer" containerID="1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e"
Sep 19 08:36:29.675235 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.675218    2559 scope.go:117] "RemoveContainer" containerID="13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2"
Sep 19 08:36:29.683087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.683070    2559 scope.go:117] "RemoveContainer" containerID="68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab"
Sep 19 08:36:29.689098 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.689076    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"]
Sep 19 08:36:29.691868 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.691844    2559 scope.go:117] "RemoveContainer" containerID="1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e"
Sep 19 08:36:29.692137 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:36:29.692119    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e\": container with ID starting with 1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e not found: ID does not exist" containerID="1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e"
Sep 19 08:36:29.692221 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.692149    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e"} err="failed to get container status \"1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e\": rpc error: code = NotFound desc = could not find container \"1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e\": container with ID starting with 1fc4935c1d7cd84553e90665a727719beea3a3ec3942115e7cb65dae28962d8e not found: ID does not exist"
Sep 19 08:36:29.692221 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.692174    2559 scope.go:117] "RemoveContainer" containerID="13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2"
Sep 19 08:36:29.692463 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.692445    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5ffdccb94c-wl6vk"]
Sep 19 08:36:29.692514 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:36:29.692481    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2\": container with ID starting with 13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2 not found: ID does not exist" containerID="13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2"
Sep 19 08:36:29.692514 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.692503    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2"} err="failed to get container status \"13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2\": rpc error: code = NotFound desc = could not find container \"13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2\": container with ID starting with 13b1d23997569e558ab6383d0711262f5749fe2347c43da714282148544062d2 not found: ID does not exist"
Sep 19 08:36:29.692581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.692519    2559 scope.go:117] "RemoveContainer" containerID="68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab"
Sep 19 08:36:29.692755 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:36:29.692740    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab\": container with ID starting with 68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab not found: ID does not exist" containerID="68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab"
Sep 19 08:36:29.692797 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.692761    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab"} err="failed to get container status \"68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab\": rpc error: code = NotFound desc = could not find container \"68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab\": container with ID starting with 68cd1dcf098db5eff7870f34eeffdd1d825ed65cc9354f1904459ce76075c4ab not found: ID does not exist"
Sep 19 08:36:29.937987 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:29.937906    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" path="/var/lib/kubelet/pods/eb10cf8e-9ab3-4539-878f-ef209aa23008/volumes"
Sep 19 08:36:30.969444 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:30.969417    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:36:31.100993 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.100948    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-cabundle-cert\") pod \"163f6283-d776-43d9-810b-e3443feccbbf\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") "
Sep 19 08:36:31.100993 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.101008    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-kube-rbac-proxy-sar-config\") pod \"163f6283-d776-43d9-810b-e3443feccbbf\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") "
Sep 19 08:36:31.101233 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.101049    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49klm\" (UniqueName: \"kubernetes.io/projected/163f6283-d776-43d9-810b-e3443feccbbf-kube-api-access-49klm\") pod \"163f6283-d776-43d9-810b-e3443feccbbf\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") "
Sep 19 08:36:31.101233 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.101082    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/163f6283-d776-43d9-810b-e3443feccbbf-kserve-provision-location\") pod \"163f6283-d776-43d9-810b-e3443feccbbf\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") "
Sep 19 08:36:31.101233 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.101134    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/163f6283-d776-43d9-810b-e3443feccbbf-proxy-tls\") pod \"163f6283-d776-43d9-810b-e3443feccbbf\" (UID: \"163f6283-d776-43d9-810b-e3443feccbbf\") "
Sep 19 08:36:31.101444 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.101418    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "163f6283-d776-43d9-810b-e3443feccbbf" (UID: "163f6283-d776-43d9-810b-e3443feccbbf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:36:31.101508 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.101436    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/163f6283-d776-43d9-810b-e3443feccbbf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "163f6283-d776-43d9-810b-e3443feccbbf" (UID: "163f6283-d776-43d9-810b-e3443feccbbf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:36:31.101508 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.101425    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "163f6283-d776-43d9-810b-e3443feccbbf" (UID: "163f6283-d776-43d9-810b-e3443feccbbf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:36:31.103242 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.103221    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163f6283-d776-43d9-810b-e3443feccbbf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "163f6283-d776-43d9-810b-e3443feccbbf" (UID: "163f6283-d776-43d9-810b-e3443feccbbf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:36:31.103339 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.103223    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163f6283-d776-43d9-810b-e3443feccbbf-kube-api-access-49klm" (OuterVolumeSpecName: "kube-api-access-49klm") pod "163f6283-d776-43d9-810b-e3443feccbbf" (UID: "163f6283-d776-43d9-810b-e3443feccbbf"). InnerVolumeSpecName "kube-api-access-49klm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:36:31.202018 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.201959    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:36:31.202018 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.202012    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/163f6283-d776-43d9-810b-e3443feccbbf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:36:31.202018 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.202027    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49klm\" (UniqueName: \"kubernetes.io/projected/163f6283-d776-43d9-810b-e3443feccbbf-kube-api-access-49klm\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:36:31.202319 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.202039    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/163f6283-d776-43d9-810b-e3443feccbbf-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:36:31.202319 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.202051    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/163f6283-d776-43d9-810b-e3443feccbbf-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:36:31.678784 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.678747    2559 generic.go:356] "Generic (PLEG): container finished" podID="163f6283-d776-43d9-810b-e3443feccbbf" containerID="908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc" exitCode=0
Sep 19 08:36:31.678967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.678807    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"
Sep 19 08:36:31.678967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.678810    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" event={"ID":"163f6283-d776-43d9-810b-e3443feccbbf","Type":"ContainerDied","Data":"908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc"}
Sep 19 08:36:31.678967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.678861    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9" event={"ID":"163f6283-d776-43d9-810b-e3443feccbbf","Type":"ContainerDied","Data":"db50e8a41b226e8eaff24392d08c01602c5ac0072da7425521b3a0ac2f730432"}
Sep 19 08:36:31.678967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.678894    2559 scope.go:117] "RemoveContainer" containerID="91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac"
Sep 19 08:36:31.693661 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.693644    2559 scope.go:117] "RemoveContainer" containerID="908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc"
Sep 19 08:36:31.704121 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.702750    2559 scope.go:117] "RemoveContainer" containerID="9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c"
Sep 19 08:36:31.706191 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.706168    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"]
Sep 19 08:36:31.709880 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.709860    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-b9c8d9798-vbnr9"]
Sep 19 08:36:31.712868 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.712853    2559 scope.go:117] "RemoveContainer" containerID="91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac"
Sep 19 08:36:31.713129 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:36:31.713111    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac\": container with ID starting with 91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac not found: ID does not exist" containerID="91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac"
Sep 19 08:36:31.713217 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.713141    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac"} err="failed to get container status \"91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac\": rpc error: code = NotFound desc = could not find container \"91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac\": container with ID starting with 91717199f0268a2f957591898a1863e48ad83ad0cffac8f8f1602b7f530084ac not found: ID does not exist"
Sep 19 08:36:31.713217 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.713165    2559 scope.go:117] "RemoveContainer" containerID="908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc"
Sep 19 08:36:31.713445 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:36:31.713425    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc\": container with ID starting with 908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc not found: ID does not exist" containerID="908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc"
Sep 19 08:36:31.713485 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.713451    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc"} err="failed to get container status \"908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc\": rpc error: code = NotFound desc = could not find container \"908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc\": container with ID starting with 908241ff6bcb6a3809c66f331807a9eb07441a4b6318bb290f1df7778f157fdc not found: ID does not exist"
Sep 19 08:36:31.713485 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.713466    2559 scope.go:117] "RemoveContainer" containerID="9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c"
Sep 19 08:36:31.713719 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:36:31.713697    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c\": container with ID starting with 9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c not found: ID does not exist" containerID="9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c"
Sep 19 08:36:31.713771 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.713725    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c"} err="failed to get container status \"9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c\": rpc error: code = NotFound desc = could not find container \"9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c\": container with ID starting with 9cd529782e11a59d05b02d50df8fab0ff4f99f0cf1a8d53f6fbcf7770abc331c not found: ID does not exist"
Sep 19 08:36:31.938210 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:31.938130    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="163f6283-d776-43d9-810b-e3443feccbbf" path="/var/lib/kubelet/pods/163f6283-d776-43d9-810b-e3443feccbbf/volumes"
Sep 19 08:36:35.656902 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.656870    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:36:35.677947 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.677910    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"]
Sep 19 08:36:35.678374 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678350    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="storage-initializer"
Sep 19 08:36:35.678374 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678375    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="storage-initializer"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678386    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="storage-initializer"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678394    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="storage-initializer"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678409    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678416    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678425    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kube-rbac-proxy"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678432    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kube-rbac-proxy"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678445    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="storage-initializer"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678453    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="storage-initializer"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678463    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kube-rbac-proxy"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678471    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kube-rbac-proxy"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678483    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678490    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678505    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678514    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678525    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kube-rbac-proxy"
Sep 19 08:36:35.678544 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678533    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kube-rbac-proxy"
Sep 19 08:36:35.679313 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678622    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kube-rbac-proxy"
Sep 19 08:36:35.679313 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678638    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kserve-container"
Sep 19 08:36:35.679313 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678646    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="173aa326-e2dc-467e-bc8a-84fa3385ee4a" containerName="kube-rbac-proxy"
Sep 19 08:36:35.679313 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678655    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kube-rbac-proxy"
Sep 19 08:36:35.679313 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678663    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="163f6283-d776-43d9-810b-e3443feccbbf" containerName="kserve-container"
Sep 19 08:36:35.679313 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.678675    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb10cf8e-9ab3-4539-878f-ef209aa23008" containerName="kserve-container"
Sep 19 08:36:35.682000 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.681974    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.684381 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.684360    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-53e8e6-kube-rbac-proxy-sar-config\""
Sep 19 08:36:35.684489 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.684352    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-53e8e6-predictor-serving-cert\""
Sep 19 08:36:35.690352 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.690326    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"]
Sep 19 08:36:35.743976 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.743941    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56kbs\" (UniqueName: \"kubernetes.io/projected/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-api-access-56kbs\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.744171 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.744016    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-cabundle-cert\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.744171 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.744060    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-rbac-proxy-sar-config\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.744171 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.744089    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb94947-e787-45bc-bc74-af5b5792d8d8-proxy-tls\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.744171 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.744120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddb94947-e787-45bc-bc74-af5b5792d8d8-kserve-provision-location\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.845335 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.845289    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56kbs\" (UniqueName: \"kubernetes.io/projected/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-api-access-56kbs\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.845540 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.845361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-cabundle-cert\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.845540 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.845389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-rbac-proxy-sar-config\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.845540 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.845418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb94947-e787-45bc-bc74-af5b5792d8d8-proxy-tls\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.845540 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.845448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddb94947-e787-45bc-bc74-af5b5792d8d8-kserve-provision-location\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.845751 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:36:35.845546    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-serving-cert: secret "isvc-primary-53e8e6-predictor-serving-cert" not found
Sep 19 08:36:35.845751 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:36:35.845618    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddb94947-e787-45bc-bc74-af5b5792d8d8-proxy-tls podName:ddb94947-e787-45bc-bc74-af5b5792d8d8 nodeName:}" failed. No retries permitted until 2026-09-19 08:36:36.345596596 +0000 UTC m=+948.756119576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ddb94947-e787-45bc-bc74-af5b5792d8d8-proxy-tls") pod "isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" (UID: "ddb94947-e787-45bc-bc74-af5b5792d8d8") : secret "isvc-primary-53e8e6-predictor-serving-cert" not found
Sep 19 08:36:35.845974 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.845950    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddb94947-e787-45bc-bc74-af5b5792d8d8-kserve-provision-location\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.846191 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.846172    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-cabundle-cert\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.846282 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.846174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-rbac-proxy-sar-config\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:35.853878 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:35.853855    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56kbs\" (UniqueName: \"kubernetes.io/projected/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-api-access-56kbs\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:36.105586 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:36.105546    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:36:36.350625 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:36.350589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb94947-e787-45bc-bc74-af5b5792d8d8-proxy-tls\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:36.353085 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:36.353062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb94947-e787-45bc-bc74-af5b5792d8d8-proxy-tls\") pod \"isvc-primary-53e8e6-predictor-869dd779c9-b9zkh\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") " pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:36.593619 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:36.593575    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:36.866018 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:36.865995    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"]
Sep 19 08:36:36.867937 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:36:36.867909    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb94947_e787_45bc_bc74_af5b5792d8d8.slice/crio-75277a6bce4834fe34b02f74e83c1631a7fe6f8ac10e2e8517c4994cc8a9d847 WatchSource:0}: Error finding container 75277a6bce4834fe34b02f74e83c1631a7fe6f8ac10e2e8517c4994cc8a9d847: Status 404 returned error can't find the container with id 75277a6bce4834fe34b02f74e83c1631a7fe6f8ac10e2e8517c4994cc8a9d847
Sep 19 08:36:37.708113 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:37.708072    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" event={"ID":"ddb94947-e787-45bc-bc74-af5b5792d8d8","Type":"ContainerStarted","Data":"4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec"}
Sep 19 08:36:37.708301 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:37.708173    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" event={"ID":"ddb94947-e787-45bc-bc74-af5b5792d8d8","Type":"ContainerStarted","Data":"75277a6bce4834fe34b02f74e83c1631a7fe6f8ac10e2e8517c4994cc8a9d847"}
Sep 19 08:36:37.939179 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:37.939151    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:36:41.728423 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:41.728388    2559 generic.go:356] "Generic (PLEG): container finished" podID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerID="4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec" exitCode=0
Sep 19 08:36:41.728824 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:41.728462    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" event={"ID":"ddb94947-e787-45bc-bc74-af5b5792d8d8","Type":"ContainerDied","Data":"4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec"}
Sep 19 08:36:42.735162 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:42.735121    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" event={"ID":"ddb94947-e787-45bc-bc74-af5b5792d8d8","Type":"ContainerStarted","Data":"50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd"}
Sep 19 08:36:42.735610 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:42.735189    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" event={"ID":"ddb94947-e787-45bc-bc74-af5b5792d8d8","Type":"ContainerStarted","Data":"aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0"}
Sep 19 08:36:42.735610 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:42.735470    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:42.735715 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:42.735613    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:42.736995 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:42.736966    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 08:36:42.754614 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:42.754565    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podStartSLOduration=3.87446062 podStartE2EDuration="7.754550153s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.880089533s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:36:42.753471063 +0000 UTC m=+955.163994067" watchObservedRunningTime="2026-09-19 08:36:42.754550153 +0000 UTC m=+955.165073154"
Sep 19 08:36:43.740272 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:43.740210    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 08:36:48.745237 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:48.745206    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:36:48.745841 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:48.745811    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 08:36:51.367048 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:51.367014    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:36:52.432317 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:52.432278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:36:58.746322 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:36:58.746277    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 08:37:01.441475 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.441385    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:01.509786 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.509744    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"]
Sep 19 08:37:01.510041 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.510012    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container" containerID="cri-o://baa42acb89061e04f1362d865abf5bcd3c785ddabea114dd2a81c80642305dca" gracePeriod=30
Sep 19 08:37:01.510127 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.510049    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kube-rbac-proxy" containerID="cri-o://ae912b1fb33be23317d6a082f9afc976cd59e139f6603bd111e6007481088022" gracePeriod=30
Sep 19 08:37:01.542615 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.542585    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:01.622625 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.622575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:01.637426 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.637399    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"]
Sep 19 08:37:01.641002 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.640983    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:01.643515 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.643494    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-271fb-kube-rbac-proxy-sar-config\""
Sep 19 08:37:01.643756 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.643733    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-271fb-predictor-serving-cert\""
Sep 19 08:37:01.652021 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.652000    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"]
Sep 19 08:37:01.660008 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.659984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vls4z\" (UniqueName: \"kubernetes.io/projected/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-api-access-vls4z\") pod \"message-dumper-raw-271fb-predictor-59f84b4df4-g27h5\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:01.660127 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.660106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-271fb-predictor-59f84b4df4-g27h5\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:01.660243 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.660160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91737d3c-3c56-45d5-bf35-e9a681146b9a-proxy-tls\") pod \"message-dumper-raw-271fb-predictor-59f84b4df4-g27h5\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:01.694460 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.694386    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:01.761461 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.761426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-271fb-predictor-59f84b4df4-g27h5\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:01.761652 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.761481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91737d3c-3c56-45d5-bf35-e9a681146b9a-proxy-tls\") pod \"message-dumper-raw-271fb-predictor-59f84b4df4-g27h5\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:01.761652 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.761506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vls4z\" (UniqueName: \"kubernetes.io/projected/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-api-access-vls4z\") pod \"message-dumper-raw-271fb-predictor-59f84b4df4-g27h5\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:01.762085 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.762066    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-271fb-predictor-59f84b4df4-g27h5\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:01.763934 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.763908    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91737d3c-3c56-45d5-bf35-e9a681146b9a-proxy-tls\") pod \"message-dumper-raw-271fb-predictor-59f84b4df4-g27h5\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:01.769761 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.769738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vls4z\" (UniqueName: \"kubernetes.io/projected/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-api-access-vls4z\") pod \"message-dumper-raw-271fb-predictor-59f84b4df4-g27h5\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") " pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:01.823552 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.823520    2559 generic.go:356] "Generic (PLEG): container finished" podID="b9d97776-bb73-4804-bb26-6566cef560b5" containerID="ae912b1fb33be23317d6a082f9afc976cd59e139f6603bd111e6007481088022" exitCode=2
Sep 19 08:37:01.823709 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.823604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" event={"ID":"b9d97776-bb73-4804-bb26-6566cef560b5","Type":"ContainerDied","Data":"ae912b1fb33be23317d6a082f9afc976cd59e139f6603bd111e6007481088022"}
Sep 19 08:37:01.952301 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:01.952186    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:02.234040 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:02.234015    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:37:02.234887 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:02.234847    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"]
Sep 19 08:37:02.831158 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:02.831123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5" event={"ID":"91737d3c-3c56-45d5-bf35-e9a681146b9a","Type":"ContainerStarted","Data":"59e4e5d6c575d9b94fde51876ceb68a444dcc0155e1fd3f99a2f13a452f4372b"}
Sep 19 08:37:03.307116 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:03.307069    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 19 08:37:03.838115 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:03.838080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5" event={"ID":"91737d3c-3c56-45d5-bf35-e9a681146b9a","Type":"ContainerStarted","Data":"b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29"}
Sep 19 08:37:03.838547 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:03.838144    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:03.838547 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:03.838169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5" event={"ID":"91737d3c-3c56-45d5-bf35-e9a681146b9a","Type":"ContainerStarted","Data":"45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb"}
Sep 19 08:37:03.838547 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:03.838191    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:03.840355 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:03.840334    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:03.857879 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:03.857824    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5" podStartSLOduration=1.610780574 podStartE2EDuration="2.857809911s" totalImagesPullingTime="1.247029337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:37:03.855390506 +0000 UTC m=+976.265913507" watchObservedRunningTime="2026-09-19 08:37:03.857809911 +0000 UTC m=+976.268332911"
Sep 19 08:37:06.854011 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:06.853981    2559 generic.go:356] "Generic (PLEG): container finished" podID="b9d97776-bb73-4804-bb26-6566cef560b5" containerID="baa42acb89061e04f1362d865abf5bcd3c785ddabea114dd2a81c80642305dca" exitCode=0
Sep 19 08:37:06.854414 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:06.854045    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" event={"ID":"b9d97776-bb73-4804-bb26-6566cef560b5","Type":"ContainerDied","Data":"baa42acb89061e04f1362d865abf5bcd3c785ddabea114dd2a81c80642305dca"}
Sep 19 08:37:06.964034 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:06.964000    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:37:07.006248 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.006175    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9d97776-bb73-4804-bb26-6566cef560b5-kserve-provision-location\") pod \"b9d97776-bb73-4804-bb26-6566cef560b5\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") "
Sep 19 08:37:07.006248 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.006220    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-kube-rbac-proxy-sar-config\") pod \"b9d97776-bb73-4804-bb26-6566cef560b5\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") "
Sep 19 08:37:07.006462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.006272    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vd6c\" (UniqueName: \"kubernetes.io/projected/b9d97776-bb73-4804-bb26-6566cef560b5-kube-api-access-6vd6c\") pod \"b9d97776-bb73-4804-bb26-6566cef560b5\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") "
Sep 19 08:37:07.006462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.006309    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d97776-bb73-4804-bb26-6566cef560b5-proxy-tls\") pod \"b9d97776-bb73-4804-bb26-6566cef560b5\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") "
Sep 19 08:37:07.006462 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.006331    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-cabundle-cert\") pod \"b9d97776-bb73-4804-bb26-6566cef560b5\" (UID: \"b9d97776-bb73-4804-bb26-6566cef560b5\") "
Sep 19 08:37:07.006629 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.006593    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d97776-bb73-4804-bb26-6566cef560b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b9d97776-bb73-4804-bb26-6566cef560b5" (UID: "b9d97776-bb73-4804-bb26-6566cef560b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:37:07.006709 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.006681    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b9d97776-bb73-4804-bb26-6566cef560b5" (UID: "b9d97776-bb73-4804-bb26-6566cef560b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:37:07.006817 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.006793    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b9d97776-bb73-4804-bb26-6566cef560b5" (UID: "b9d97776-bb73-4804-bb26-6566cef560b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:37:07.008516 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.008492    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d97776-bb73-4804-bb26-6566cef560b5-kube-api-access-6vd6c" (OuterVolumeSpecName: "kube-api-access-6vd6c") pod "b9d97776-bb73-4804-bb26-6566cef560b5" (UID: "b9d97776-bb73-4804-bb26-6566cef560b5"). InnerVolumeSpecName "kube-api-access-6vd6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:37:07.008580 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.008559    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d97776-bb73-4804-bb26-6566cef560b5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b9d97776-bb73-4804-bb26-6566cef560b5" (UID: "b9d97776-bb73-4804-bb26-6566cef560b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:37:07.107215 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.107184    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vd6c\" (UniqueName: \"kubernetes.io/projected/b9d97776-bb73-4804-bb26-6566cef560b5-kube-api-access-6vd6c\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:37:07.107215 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.107211    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b9d97776-bb73-4804-bb26-6566cef560b5-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:37:07.107215 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.107221    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:37:07.107468 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.107229    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b9d97776-bb73-4804-bb26-6566cef560b5-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:37:07.107468 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.107238    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b9d97776-bb73-4804-bb26-6566cef560b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:37:07.860415 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.860381    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"
Sep 19 08:37:07.860415 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.860398    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2" event={"ID":"b9d97776-bb73-4804-bb26-6566cef560b5","Type":"ContainerDied","Data":"069a3a050340735cce415ce224b86dc4e844228b9666aeb216ded5bde29394ef"}
Sep 19 08:37:07.860888 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.860480    2559 scope.go:117] "RemoveContainer" containerID="ae912b1fb33be23317d6a082f9afc976cd59e139f6603bd111e6007481088022"
Sep 19 08:37:07.871285 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.871247    2559 scope.go:117] "RemoveContainer" containerID="baa42acb89061e04f1362d865abf5bcd3c785ddabea114dd2a81c80642305dca"
Sep 19 08:37:07.879372 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.879355    2559 scope.go:117] "RemoveContainer" containerID="55c40e24a1e9904f1e9d1d0677a075d6a923e6b8eccba0d2c429c29e175bcafa"
Sep 19 08:37:07.886148 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.886125    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"]
Sep 19 08:37:07.889771 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.889747    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-caf7a-predictor-775d95fcd6-jn8r2"]
Sep 19 08:37:07.937968 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:07.937928    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" path="/var/lib/kubelet/pods/b9d97776-bb73-4804-bb26-6566cef560b5/volumes"
Sep 19 08:37:08.746407 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:08.746350    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 08:37:10.851351 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:10.851322    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:37:11.604086 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.604046    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:11.651471 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.651421    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"]
Sep 19 08:37:11.651914 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.651889    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container"
Sep 19 08:37:11.651914 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.651912    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container"
Sep 19 08:37:11.651914 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.651921    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kube-rbac-proxy"
Sep 19 08:37:11.652176 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.651926    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kube-rbac-proxy"
Sep 19 08:37:11.652176 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.651959    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="storage-initializer"
Sep 19 08:37:11.652176 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.651968    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="storage-initializer"
Sep 19 08:37:11.652176 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.652038    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kube-rbac-proxy"
Sep 19 08:37:11.652176 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.652049    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d97776-bb73-4804-bb26-6566cef560b5" containerName="kserve-container"
Sep 19 08:37:11.656920 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.656897    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.659500 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.659471    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-271fb-predictor-serving-cert\""
Sep 19 08:37:11.659500 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.659477    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-271fb-kube-rbac-proxy-sar-config\""
Sep 19 08:37:11.666279 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.666237    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"]
Sep 19 08:37:11.751763 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.751723    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.751943 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.751779    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kserve-provision-location\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.751943 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.751810    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-cabundle-cert\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.751943 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.751858    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhx6\" (UniqueName: \"kubernetes.io/projected/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-api-access-vbhx6\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.751943 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.751918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a9c808d-8068-4f10-ab87-2cd9099abb6e-proxy-tls\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.776999 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.776966    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:11.853060 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.853020    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.853623 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.853066    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kserve-provision-location\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.853623 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.853090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-cabundle-cert\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.853623 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.853122    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhx6\" (UniqueName: \"kubernetes.io/projected/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-api-access-vbhx6\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.853623 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.853160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a9c808d-8068-4f10-ab87-2cd9099abb6e-proxy-tls\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.853623 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.853535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kserve-provision-location\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.853936 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.853811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-cabundle-cert\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.854000 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.853980    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.855745 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.855702    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a9c808d-8068-4f10-ab87-2cd9099abb6e-proxy-tls\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.861782 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.861757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhx6\" (UniqueName: \"kubernetes.io/projected/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-api-access-vbhx6\") pod \"isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:11.969807 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:11.969770    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:12.246058 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:12.246034    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"]
Sep 19 08:37:12.248575 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:37:12.248547    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9c808d_8068_4f10_ab87_2cd9099abb6e.slice/crio-188a482914aa0f24d455fa65d07045a4abd0164181c77d0bb860fe60bcc7a6c2 WatchSource:0}: Error finding container 188a482914aa0f24d455fa65d07045a4abd0164181c77d0bb860fe60bcc7a6c2: Status 404 returned error can't find the container with id 188a482914aa0f24d455fa65d07045a4abd0164181c77d0bb860fe60bcc7a6c2
Sep 19 08:37:12.886388 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:12.886346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" event={"ID":"3a9c808d-8068-4f10-ab87-2cd9099abb6e","Type":"ContainerStarted","Data":"f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8"}
Sep 19 08:37:12.886768 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:12.886418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" event={"ID":"3a9c808d-8068-4f10-ab87-2cd9099abb6e","Type":"ContainerStarted","Data":"188a482914aa0f24d455fa65d07045a4abd0164181c77d0bb860fe60bcc7a6c2"}
Sep 19 08:37:16.907145 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:16.907107    2559 generic.go:356] "Generic (PLEG): container finished" podID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerID="f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8" exitCode=0
Sep 19 08:37:16.907581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:16.907178    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" event={"ID":"3a9c808d-8068-4f10-ab87-2cd9099abb6e","Type":"ContainerDied","Data":"f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8"}
Sep 19 08:37:17.915471 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:17.915438    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" event={"ID":"3a9c808d-8068-4f10-ab87-2cd9099abb6e","Type":"ContainerStarted","Data":"d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf"}
Sep 19 08:37:17.915843 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:17.915506    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" event={"ID":"3a9c808d-8068-4f10-ab87-2cd9099abb6e","Type":"ContainerStarted","Data":"13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8"}
Sep 19 08:37:17.915843 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:17.915531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" event={"ID":"3a9c808d-8068-4f10-ab87-2cd9099abb6e","Type":"ContainerStarted","Data":"a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6"}
Sep 19 08:37:17.915843 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:17.915834    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:17.915966 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:17.915891    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:17.915966 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:17.915938    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:17.917575 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:17.917543    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:37:17.918218 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:17.918191    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:17.937746 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:17.937701    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podStartSLOduration=3.1627032330000002 podStartE2EDuration="6.937685209s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.774981976s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:37:17.935104159 +0000 UTC m=+990.345627159" watchObservedRunningTime="2026-09-19 08:37:17.937685209 +0000 UTC m=+990.348208270"
Sep 19 08:37:18.746216 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:18.746171    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 08:37:18.921349 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:18.921309    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:37:18.921870 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:18.921845    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:20.175463 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:20.175422    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:23.926346 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:23.926317    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:37:23.926973 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:23.926941    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:37:23.927321 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:23.927290    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:28.745865 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:28.745811    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 08:37:32.672549 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:32.672507    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:32.727967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:32.727932    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:37:33.927362 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:33.927318    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:37:33.927817 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:33.927796    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:38.745838 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:38.745796    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 08:37:43.926952 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:43.926902    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:37:43.927405 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:43.927380    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:48.746265 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:48.746206    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 08:37:53.927486 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:53.927433    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:37:53.927979 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:53.927954    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:37:56.932922 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:37:56.932878    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 19 08:38:03.927838 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:03.927785    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:38:03.928409 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:03.928294    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:06.933435 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:06.933402    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:38:13.927374 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:13.927337    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:38:13.927920 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:13.927899    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:23.927665 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:23.927614    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:38:23.928096 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:23.928012    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:29.935141 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:29.935087    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:38:29.935608 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:29.935423    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:30.839989 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:30.839942    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:31.050661 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:31.050616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:39.938656 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:39.938629    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:38:39.939380 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:39.938708    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:38:46.907748 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:46.907707    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"]
Sep 19 08:38:46.908332 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:46.908024    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container" containerID="cri-o://aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0" gracePeriod=30
Sep 19 08:38:46.908332 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:46.908100    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kube-rbac-proxy" containerID="cri-o://50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd" gracePeriod=30
Sep 19 08:38:46.909708 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:46.909681    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:46.967548 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:46.967495    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:47.102907 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.102877    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-271fb-predictor-59f84b4df4-g27h5_91737d3c-3c56-45d5-bf35-e9a681146b9a/kserve-container/0.log"
Sep 19 08:38:47.257868 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.257753    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"]
Sep 19 08:38:47.258190 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.258139    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" containerID="cri-o://a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6" gracePeriod=30
Sep 19 08:38:47.258190 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.258164    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" containerID="cri-o://d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf" gracePeriod=30
Sep 19 08:38:47.258396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.258190    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kube-rbac-proxy" containerID="cri-o://13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8" gracePeriod=30
Sep 19 08:38:47.277197 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.277169    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:47.285146 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.285121    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"]
Sep 19 08:38:47.285406 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.285374    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5" podUID="91737d3c-3c56-45d5-bf35-e9a681146b9a" containerName="kserve-container" containerID="cri-o://45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb" gracePeriod=30
Sep 19 08:38:47.285543 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.285409    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5" podUID="91737d3c-3c56-45d5-bf35-e9a681146b9a" containerName="kube-rbac-proxy" containerID="cri-o://b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29" gracePeriod=30
Sep 19 08:38:47.356233 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.356198    2559 generic.go:356] "Generic (PLEG): container finished" podID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerID="50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd" exitCode=2
Sep 19 08:38:47.356431 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.356229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" event={"ID":"ddb94947-e787-45bc-bc74-af5b5792d8d8","Type":"ContainerDied","Data":"50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd"}
Sep 19 08:38:47.378449 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.378413    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:47.440757 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.440710    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:47.538464 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.538433    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:38:47.586586 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.586551    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:47.611648 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.611617    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vls4z\" (UniqueName: \"kubernetes.io/projected/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-api-access-vls4z\") pod \"91737d3c-3c56-45d5-bf35-e9a681146b9a\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") "
Sep 19 08:38:47.611813 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.611731    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-rbac-proxy-sar-config\") pod \"91737d3c-3c56-45d5-bf35-e9a681146b9a\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") "
Sep 19 08:38:47.611813 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.611798    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91737d3c-3c56-45d5-bf35-e9a681146b9a-proxy-tls\") pod \"91737d3c-3c56-45d5-bf35-e9a681146b9a\" (UID: \"91737d3c-3c56-45d5-bf35-e9a681146b9a\") "
Sep 19 08:38:47.612150 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.612124    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91737d3c-3c56-45d5-bf35-e9a681146b9a" (UID: "91737d3c-3c56-45d5-bf35-e9a681146b9a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:38:47.613746 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.613721    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-api-access-vls4z" (OuterVolumeSpecName: "kube-api-access-vls4z") pod "91737d3c-3c56-45d5-bf35-e9a681146b9a" (UID: "91737d3c-3c56-45d5-bf35-e9a681146b9a"). InnerVolumeSpecName "kube-api-access-vls4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:38:47.613882 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.613860    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91737d3c-3c56-45d5-bf35-e9a681146b9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91737d3c-3c56-45d5-bf35-e9a681146b9a" (UID: "91737d3c-3c56-45d5-bf35-e9a681146b9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:38:47.712485 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.712441    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:38:47.712485 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.712478    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91737d3c-3c56-45d5-bf35-e9a681146b9a-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:38:47.712485 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:47.712494    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vls4z\" (UniqueName: \"kubernetes.io/projected/91737d3c-3c56-45d5-bf35-e9a681146b9a-kube-api-access-vls4z\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:38:48.363326 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.363292    2559 generic.go:356] "Generic (PLEG): container finished" podID="91737d3c-3c56-45d5-bf35-e9a681146b9a" containerID="b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29" exitCode=2
Sep 19 08:38:48.363326 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.363318    2559 generic.go:356] "Generic (PLEG): container finished" podID="91737d3c-3c56-45d5-bf35-e9a681146b9a" containerID="45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb" exitCode=2
Sep 19 08:38:48.363869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.363359    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"
Sep 19 08:38:48.363869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.363377    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5" event={"ID":"91737d3c-3c56-45d5-bf35-e9a681146b9a","Type":"ContainerDied","Data":"b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29"}
Sep 19 08:38:48.363869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.363436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5" event={"ID":"91737d3c-3c56-45d5-bf35-e9a681146b9a","Type":"ContainerDied","Data":"45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb"}
Sep 19 08:38:48.363869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.363459    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5" event={"ID":"91737d3c-3c56-45d5-bf35-e9a681146b9a","Type":"ContainerDied","Data":"59e4e5d6c575d9b94fde51876ceb68a444dcc0155e1fd3f99a2f13a452f4372b"}
Sep 19 08:38:48.363869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.363487    2559 scope.go:117] "RemoveContainer" containerID="b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29"
Sep 19 08:38:48.366380 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.366353    2559 generic.go:356] "Generic (PLEG): container finished" podID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerID="13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8" exitCode=2
Sep 19 08:38:48.366478 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.366416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" event={"ID":"3a9c808d-8068-4f10-ab87-2cd9099abb6e","Type":"ContainerDied","Data":"13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8"}
Sep 19 08:38:48.372172 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.372154    2559 scope.go:117] "RemoveContainer" containerID="45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb"
Sep 19 08:38:48.380114 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.380097    2559 scope.go:117] "RemoveContainer" containerID="b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29"
Sep 19 08:38:48.380413 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:38:48.380382    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29\": container with ID starting with b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29 not found: ID does not exist" containerID="b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29"
Sep 19 08:38:48.380503 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.380419    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29"} err="failed to get container status \"b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29\": rpc error: code = NotFound desc = could not find container \"b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29\": container with ID starting with b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29 not found: ID does not exist"
Sep 19 08:38:48.380503 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.380442    2559 scope.go:117] "RemoveContainer" containerID="45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb"
Sep 19 08:38:48.380679 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:38:48.380658    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb\": container with ID starting with 45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb not found: ID does not exist" containerID="45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb"
Sep 19 08:38:48.380720 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.380684    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb"} err="failed to get container status \"45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb\": rpc error: code = NotFound desc = could not find container \"45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb\": container with ID starting with 45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb not found: ID does not exist"
Sep 19 08:38:48.380720 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.380698    2559 scope.go:117] "RemoveContainer" containerID="b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29"
Sep 19 08:38:48.380884 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.380869    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29"} err="failed to get container status \"b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29\": rpc error: code = NotFound desc = could not find container \"b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29\": container with ID starting with b03e48ee25692ff0925ab9a9ce0503a487f0c8d4e9e3a0c46293576d24dd2a29 not found: ID does not exist"
Sep 19 08:38:48.380932 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.380884    2559 scope.go:117] "RemoveContainer" containerID="45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb"
Sep 19 08:38:48.381094 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.381078    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb"} err="failed to get container status \"45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb\": rpc error: code = NotFound desc = could not find container \"45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb\": container with ID starting with 45b0e7a48054d7912b1bac85e98394638bb7e9098b432377733b8690ed7918eb not found: ID does not exist"
Sep 19 08:38:48.381628 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.381611    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"]
Sep 19 08:38:48.385263 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.385234    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-271fb-predictor-59f84b4df4-g27h5"]
Sep 19 08:38:48.740731 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.740625    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 19 08:38:48.921643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:48.921600    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 19 08:38:49.935174 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:49.935119    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:38:49.935673 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:49.935558    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:38:49.938450 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:49.938431    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91737d3c-3c56-45d5-bf35-e9a681146b9a" path="/var/lib/kubelet/pods/91737d3c-3c56-45d5-bf35-e9a681146b9a/volumes"
Sep 19 08:38:52.073956 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.073918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:52.171607 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.171567    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:38:52.363479 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.363455    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:38:52.389643 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.389613    2559 generic.go:356] "Generic (PLEG): container finished" podID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerID="aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0" exitCode=0
Sep 19 08:38:52.389815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.389672    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"
Sep 19 08:38:52.389815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.389692    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" event={"ID":"ddb94947-e787-45bc-bc74-af5b5792d8d8","Type":"ContainerDied","Data":"aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0"}
Sep 19 08:38:52.389815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.389753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh" event={"ID":"ddb94947-e787-45bc-bc74-af5b5792d8d8","Type":"ContainerDied","Data":"75277a6bce4834fe34b02f74e83c1631a7fe6f8ac10e2e8517c4994cc8a9d847"}
Sep 19 08:38:52.389815 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.389782    2559 scope.go:117] "RemoveContainer" containerID="50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd"
Sep 19 08:38:52.399036 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.399019    2559 scope.go:117] "RemoveContainer" containerID="aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0"
Sep 19 08:38:52.406830 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.406813    2559 scope.go:117] "RemoveContainer" containerID="4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec"
Sep 19 08:38:52.414348 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.414332    2559 scope.go:117] "RemoveContainer" containerID="50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd"
Sep 19 08:38:52.414607 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:38:52.414588    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd\": container with ID starting with 50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd not found: ID does not exist" containerID="50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd"
Sep 19 08:38:52.414653 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.414618    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd"} err="failed to get container status \"50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd\": rpc error: code = NotFound desc = could not find container \"50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd\": container with ID starting with 50bef1d02753df83635355a0b5e94c93642d94cbf3497e47dea2a41694ef13fd not found: ID does not exist"
Sep 19 08:38:52.414653 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.414639    2559 scope.go:117] "RemoveContainer" containerID="aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0"
Sep 19 08:38:52.414871 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:38:52.414853    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0\": container with ID starting with aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0 not found: ID does not exist" containerID="aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0"
Sep 19 08:38:52.414941 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.414881    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0"} err="failed to get container status \"aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0\": rpc error: code = NotFound desc = could not find container \"aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0\": container with ID starting with aa8b59894468ca31d22ede87ffeb5b0de23794463a7569fa7e45dced653bcde0 not found: ID does not exist"
Sep 19 08:38:52.414941 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.414905    2559 scope.go:117] "RemoveContainer" containerID="4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec"
Sep 19 08:38:52.415150 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:38:52.415132    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec\": container with ID starting with 4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec not found: ID does not exist" containerID="4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec"
Sep 19 08:38:52.415200 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.415156    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec"} err="failed to get container status \"4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec\": rpc error: code = NotFound desc = could not find container \"4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec\": container with ID starting with 4f06fec46cdb9a5b939ac9b912e2b11d86fd0dfdd33b6b2091e8a062526ca2ec not found: ID does not exist"
Sep 19 08:38:52.455184 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.455155    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddb94947-e787-45bc-bc74-af5b5792d8d8-kserve-provision-location\") pod \"ddb94947-e787-45bc-bc74-af5b5792d8d8\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") "
Sep 19 08:38:52.455339 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.455192    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb94947-e787-45bc-bc74-af5b5792d8d8-proxy-tls\") pod \"ddb94947-e787-45bc-bc74-af5b5792d8d8\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") "
Sep 19 08:38:52.455339 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.455224    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56kbs\" (UniqueName: \"kubernetes.io/projected/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-api-access-56kbs\") pod \"ddb94947-e787-45bc-bc74-af5b5792d8d8\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") "
Sep 19 08:38:52.455339 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.455295    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-rbac-proxy-sar-config\") pod \"ddb94947-e787-45bc-bc74-af5b5792d8d8\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") "
Sep 19 08:38:52.455339 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.455317    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-cabundle-cert\") pod \"ddb94947-e787-45bc-bc74-af5b5792d8d8\" (UID: \"ddb94947-e787-45bc-bc74-af5b5792d8d8\") "
Sep 19 08:38:52.455566 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.455549    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb94947-e787-45bc-bc74-af5b5792d8d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ddb94947-e787-45bc-bc74-af5b5792d8d8" (UID: "ddb94947-e787-45bc-bc74-af5b5792d8d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:38:52.455682 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.455661    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ddb94947-e787-45bc-bc74-af5b5792d8d8" (UID: "ddb94947-e787-45bc-bc74-af5b5792d8d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:38:52.455736 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.455711    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ddb94947-e787-45bc-bc74-af5b5792d8d8" (UID: "ddb94947-e787-45bc-bc74-af5b5792d8d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:38:52.457350 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.457320    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-api-access-56kbs" (OuterVolumeSpecName: "kube-api-access-56kbs") pod "ddb94947-e787-45bc-bc74-af5b5792d8d8" (UID: "ddb94947-e787-45bc-bc74-af5b5792d8d8"). InnerVolumeSpecName "kube-api-access-56kbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:38:52.457441 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.457324    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb94947-e787-45bc-bc74-af5b5792d8d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ddb94947-e787-45bc-bc74-af5b5792d8d8" (UID: "ddb94947-e787-45bc-bc74-af5b5792d8d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:38:52.556037 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.556011    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddb94947-e787-45bc-bc74-af5b5792d8d8-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:38:52.556139 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.556042    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb94947-e787-45bc-bc74-af5b5792d8d8-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:38:52.556139 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.556057    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56kbs\" (UniqueName: \"kubernetes.io/projected/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-api-access-56kbs\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:38:52.556139 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.556069    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:38:52.556139 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.556082    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddb94947-e787-45bc-bc74-af5b5792d8d8-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:38:52.714611 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.714579    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"]
Sep 19 08:38:52.717930 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:52.717906    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-53e8e6-predictor-869dd779c9-b9zkh"]
Sep 19 08:38:53.399242 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:53.399209    2559 generic.go:356] "Generic (PLEG): container finished" podID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerID="a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6" exitCode=0
Sep 19 08:38:53.399662 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:53.399289    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" event={"ID":"3a9c808d-8068-4f10-ab87-2cd9099abb6e","Type":"ContainerDied","Data":"a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6"}
Sep 19 08:38:53.921636 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:53.921589    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 19 08:38:53.938059 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:53.938030    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" path="/var/lib/kubelet/pods/ddb94947-e787-45bc-bc74-af5b5792d8d8/volumes"
Sep 19 08:38:58.922456 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:58.922414    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 19 08:38:58.922860 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:58.922562    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:38:59.935154 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:59.935114    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:38:59.935632 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:38:59.935543    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:39:02.096839 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:02.096806    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:39:03.921860 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:03.921814    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 19 08:39:07.268756 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.268719    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:39:07.297275 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297220    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"]
Sep 19 08:39:07.297652 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297636    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91737d3c-3c56-45d5-bf35-e9a681146b9a" containerName="kserve-container"
Sep 19 08:39:07.297652 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297654    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="91737d3c-3c56-45d5-bf35-e9a681146b9a" containerName="kserve-container"
Sep 19 08:39:07.297806 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297672    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container"
Sep 19 08:39:07.297806 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297678    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container"
Sep 19 08:39:07.297806 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297696    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kube-rbac-proxy"
Sep 19 08:39:07.297806 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297704    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kube-rbac-proxy"
Sep 19 08:39:07.297806 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297715    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91737d3c-3c56-45d5-bf35-e9a681146b9a" containerName="kube-rbac-proxy"
Sep 19 08:39:07.297806 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297723    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="91737d3c-3c56-45d5-bf35-e9a681146b9a" containerName="kube-rbac-proxy"
Sep 19 08:39:07.297806 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297734    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="storage-initializer"
Sep 19 08:39:07.297806 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297740    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="storage-initializer"
Sep 19 08:39:07.297806 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297810    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kserve-container"
Sep 19 08:39:07.298303 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297818    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="91737d3c-3c56-45d5-bf35-e9a681146b9a" containerName="kserve-container"
Sep 19 08:39:07.298303 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297825    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddb94947-e787-45bc-bc74-af5b5792d8d8" containerName="kube-rbac-proxy"
Sep 19 08:39:07.298303 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.297834    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="91737d3c-3c56-45d5-bf35-e9a681146b9a" containerName="kube-rbac-proxy"
Sep 19 08:39:07.301099 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.301073    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.303450 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.303424    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-655f6-predictor-serving-cert\""
Sep 19 08:39:07.303639 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.303618    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-655f6-kube-rbac-proxy-sar-config\""
Sep 19 08:39:07.311836 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.311809    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"]
Sep 19 08:39:07.395889 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.395852    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-cabundle-cert\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.395889 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.395892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.396114 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.395981    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kserve-provision-location\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.396114 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.396059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-proxy-tls\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.396187 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.396114    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktv4\" (UniqueName: \"kubernetes.io/projected/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-api-access-gktv4\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.494718 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.494684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:39:07.496879 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.496861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kserve-provision-location\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.496941 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.496897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-proxy-tls\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.496941 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.496936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktv4\" (UniqueName: \"kubernetes.io/projected/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-api-access-gktv4\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.497014 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.496969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-cabundle-cert\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.497014 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.496990    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.497367 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.497341    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kserve-provision-location\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.497661 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.497636    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-cabundle-cert\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.497725 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.497661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.499532 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.499509    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-proxy-tls\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.505471 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.505449    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktv4\" (UniqueName: \"kubernetes.io/projected/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-api-access-gktv4\") pod \"raw-sklearn-655f6-predictor-f56454574-585d8\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") " pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.613052 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.613010    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:07.890286 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:07.890240    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"]
Sep 19 08:39:07.891947 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:39:07.891917    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded93f295_4ccf_4c18_b7ae_55a25b55ebdd.slice/crio-4e40a4d694060a7862c864f8d9c179856a799f0136160373cb483f24b2b76a09 WatchSource:0}: Error finding container 4e40a4d694060a7862c864f8d9c179856a799f0136160373cb483f24b2b76a09: Status 404 returned error can't find the container with id 4e40a4d694060a7862c864f8d9c179856a799f0136160373cb483f24b2b76a09
Sep 19 08:39:08.469408 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:08.469369    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" event={"ID":"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd","Type":"ContainerStarted","Data":"d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7"}
Sep 19 08:39:08.469922 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:08.469436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" event={"ID":"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd","Type":"ContainerStarted","Data":"4e40a4d694060a7862c864f8d9c179856a799f0136160373cb483f24b2b76a09"}
Sep 19 08:39:08.921801 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:08.921761    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 19 08:39:09.935056 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:09.935008    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 19 08:39:09.935535 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:09.935393    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:39:09.938099 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:09.938082    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:39:09.938183 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:09.938170    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:39:12.489202 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:12.489157    2559 generic.go:356] "Generic (PLEG): container finished" podID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerID="d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7" exitCode=0
Sep 19 08:39:12.489744 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:12.489217    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" event={"ID":"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd","Type":"ContainerDied","Data":"d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7"}
Sep 19 08:39:13.496317 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:13.496279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" event={"ID":"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd","Type":"ContainerStarted","Data":"3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc"}
Sep 19 08:39:13.496757 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:13.496348    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" event={"ID":"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd","Type":"ContainerStarted","Data":"ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565"}
Sep 19 08:39:13.496757 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:13.496567    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:13.519785 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:13.519738    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podStartSLOduration=2.560044049 podStartE2EDuration="6.519721241s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.959677192s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:39:13.51701537 +0000 UTC m=+1105.927538370" watchObservedRunningTime="2026-09-19 08:39:13.519721241 +0000 UTC m=+1105.930244241"
Sep 19 08:39:13.922231 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:13.922186    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 19 08:39:14.501777 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:14.501746    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:14.503187 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:14.503162    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 19 08:39:15.506976 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:15.506935    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 19 08:39:17.417039 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.417014    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:39:17.494793 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.494766    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-rbac-proxy-sar-config\") pod \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") "
Sep 19 08:39:17.494977 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.494799    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kserve-provision-location\") pod \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") "
Sep 19 08:39:17.494977 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.494880    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbhx6\" (UniqueName: \"kubernetes.io/projected/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-api-access-vbhx6\") pod \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") "
Sep 19 08:39:17.494977 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.494905    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a9c808d-8068-4f10-ab87-2cd9099abb6e-proxy-tls\") pod \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") "
Sep 19 08:39:17.494977 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.494937    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-cabundle-cert\") pod \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\" (UID: \"3a9c808d-8068-4f10-ab87-2cd9099abb6e\") "
Sep 19 08:39:17.495176 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.495148    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a9c808d-8068-4f10-ab87-2cd9099abb6e" (UID: "3a9c808d-8068-4f10-ab87-2cd9099abb6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:39:17.495234 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.495189    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a9c808d-8068-4f10-ab87-2cd9099abb6e" (UID: "3a9c808d-8068-4f10-ab87-2cd9099abb6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:39:17.495371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.495352    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:39:17.495371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.495368    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:39:17.495530 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.495464    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a9c808d-8068-4f10-ab87-2cd9099abb6e" (UID: "3a9c808d-8068-4f10-ab87-2cd9099abb6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:39:17.496979 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.496955    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-api-access-vbhx6" (OuterVolumeSpecName: "kube-api-access-vbhx6") pod "3a9c808d-8068-4f10-ab87-2cd9099abb6e" (UID: "3a9c808d-8068-4f10-ab87-2cd9099abb6e"). InnerVolumeSpecName "kube-api-access-vbhx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:39:17.496979 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.496963    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a9c808d-8068-4f10-ab87-2cd9099abb6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a9c808d-8068-4f10-ab87-2cd9099abb6e" (UID: "3a9c808d-8068-4f10-ab87-2cd9099abb6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:39:17.519377 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.519311    2559 generic.go:356] "Generic (PLEG): container finished" podID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerID="d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf" exitCode=0
Sep 19 08:39:17.519377 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.519373    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"
Sep 19 08:39:17.519536 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.519397    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" event={"ID":"3a9c808d-8068-4f10-ab87-2cd9099abb6e","Type":"ContainerDied","Data":"d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf"}
Sep 19 08:39:17.519536 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.519461    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj" event={"ID":"3a9c808d-8068-4f10-ab87-2cd9099abb6e","Type":"ContainerDied","Data":"188a482914aa0f24d455fa65d07045a4abd0164181c77d0bb860fe60bcc7a6c2"}
Sep 19 08:39:17.519536 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.519488    2559 scope.go:117] "RemoveContainer" containerID="d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf"
Sep 19 08:39:17.527986 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.527967    2559 scope.go:117] "RemoveContainer" containerID="13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8"
Sep 19 08:39:17.535545 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.535529    2559 scope.go:117] "RemoveContainer" containerID="a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6"
Sep 19 08:39:17.541466 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.541445    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"]
Sep 19 08:39:17.543670 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.543642    2559 scope.go:117] "RemoveContainer" containerID="f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8"
Sep 19 08:39:17.548732 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.548712    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-271fb-predictor-59cb477f85-q5vhj"]
Sep 19 08:39:17.552213 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.552193    2559 scope.go:117] "RemoveContainer" containerID="d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf"
Sep 19 08:39:17.552547 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:39:17.552530    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf\": container with ID starting with d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf not found: ID does not exist" containerID="d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf"
Sep 19 08:39:17.552612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.552556    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf"} err="failed to get container status \"d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf\": rpc error: code = NotFound desc = could not find container \"d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf\": container with ID starting with d2cc024ca45810107e1dcd5255e306a3747228a8edf2f614c2cf06285be85faf not found: ID does not exist"
Sep 19 08:39:17.552612 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.552573    2559 scope.go:117] "RemoveContainer" containerID="13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8"
Sep 19 08:39:17.552832 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:39:17.552815    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8\": container with ID starting with 13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8 not found: ID does not exist" containerID="13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8"
Sep 19 08:39:17.552875 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.552837    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8"} err="failed to get container status \"13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8\": rpc error: code = NotFound desc = could not find container \"13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8\": container with ID starting with 13312d99f358bf395c6883ca9bc3c7ba651d67404f9bae5ad7701aa4a66271a8 not found: ID does not exist"
Sep 19 08:39:17.552875 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.552849    2559 scope.go:117] "RemoveContainer" containerID="a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6"
Sep 19 08:39:17.553113 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:39:17.553095    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6\": container with ID starting with a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6 not found: ID does not exist" containerID="a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6"
Sep 19 08:39:17.553175 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.553130    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6"} err="failed to get container status \"a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6\": rpc error: code = NotFound desc = could not find container \"a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6\": container with ID starting with a72714e73050544c1b7d82fe34a1f3ea8ad5c10668ab1e6915f0834416cf84f6 not found: ID does not exist"
Sep 19 08:39:17.553175 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.553153    2559 scope.go:117] "RemoveContainer" containerID="f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8"
Sep 19 08:39:17.553418 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:39:17.553399    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8\": container with ID starting with f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8 not found: ID does not exist" containerID="f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8"
Sep 19 08:39:17.553496 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.553418    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8"} err="failed to get container status \"f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8\": rpc error: code = NotFound desc = could not find container \"f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8\": container with ID starting with f3e2944224bfc1bc0488447e66c3183290b5b43820cc299faee999cb4576e2e8 not found: ID does not exist"
Sep 19 08:39:17.595982 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.595945    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbhx6\" (UniqueName: \"kubernetes.io/projected/3a9c808d-8068-4f10-ab87-2cd9099abb6e-kube-api-access-vbhx6\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:39:17.595982 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.595980    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a9c808d-8068-4f10-ab87-2cd9099abb6e-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:39:17.595982 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.595992    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a9c808d-8068-4f10-ab87-2cd9099abb6e-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:39:17.939168 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:17.939135    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" path="/var/lib/kubelet/pods/3a9c808d-8068-4f10-ab87-2cd9099abb6e/volumes"
Sep 19 08:39:20.512244 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:20.512211    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:39:20.512750 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:20.512724    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 19 08:39:30.513068 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:30.513026    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 19 08:39:34.213744 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:34.213702    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:39:34.265332 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:34.265302    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:39:40.513024 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:40.512981    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 19 08:39:50.512968 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:50.512928    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 19 08:39:56.424827 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:39:56.424783    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:00.513496 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:00.513456    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 19 08:40:05.635670 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:05.635628    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:10.513562 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:10.513523    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 19 08:40:20.513545 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:20.513504    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 19 08:40:27.932775 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:27.932733    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 19 08:40:37.938159 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:37.938131    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:40:47.905937 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:47.905898    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"]
Sep 19 08:40:47.906482 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:47.906249    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container" containerID="cri-o://ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565" gracePeriod=30
Sep 19 08:40:47.906482 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:47.906316    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kube-rbac-proxy" containerID="cri-o://3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc" gracePeriod=30
Sep 19 08:40:47.926747 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:47.926724    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:48.959680 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:48.959645    2559 generic.go:356] "Generic (PLEG): container finished" podID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerID="3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc" exitCode=2
Sep 19 08:40:48.959680 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:48.959687    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" event={"ID":"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd","Type":"ContainerDied","Data":"3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc"}
Sep 19 08:40:50.508013 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:50.507970    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 19 08:40:53.025712 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.025638    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:53.076794 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.076758    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"]
Sep 19 08:40:53.077126 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.077114    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kube-rbac-proxy"
Sep 19 08:40:53.077188 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.077128    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kube-rbac-proxy"
Sep 19 08:40:53.077188 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.077143    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container"
Sep 19 08:40:53.077188 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.077148    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container"
Sep 19 08:40:53.077188 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.077155    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent"
Sep 19 08:40:53.077188 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.077160    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent"
Sep 19 08:40:53.077188 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.077176    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="storage-initializer"
Sep 19 08:40:53.077188 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.077182    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="storage-initializer"
Sep 19 08:40:53.077476 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.077237    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kube-rbac-proxy"
Sep 19 08:40:53.077476 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.077248    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="agent"
Sep 19 08:40:53.077476 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.077267    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a9c808d-8068-4f10-ab87-2cd9099abb6e" containerName="kserve-container"
Sep 19 08:40:53.080519 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.080498    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.083032 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.083013    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-44db7-predictor-serving-cert\""
Sep 19 08:40:53.083032 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.083026    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-44db7-kube-rbac-proxy-sar-config\""
Sep 19 08:40:53.091920 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.091897    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"]
Sep 19 08:40:53.163232 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.163203    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-cabundle-cert\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.163369 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.163264    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfkqg\" (UniqueName: \"kubernetes.io/projected/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-api-access-gfkqg\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.163369 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.163320    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e7b273-1bad-4df6-b60d-46ed7b92f610-proxy-tls\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.163444 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.163380    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kserve-provision-location\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.163444 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.163414    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.264710 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.264676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kserve-provision-location\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.264710 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.264716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.264974 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.264749    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-cabundle-cert\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.264974 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.264862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfkqg\" (UniqueName: \"kubernetes.io/projected/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-api-access-gfkqg\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.264974 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.264930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e7b273-1bad-4df6-b60d-46ed7b92f610-proxy-tls\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.265299 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.265241    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kserve-provision-location\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.265522 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.265501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-cabundle-cert\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.265576 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.265505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.267423 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.267404    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e7b273-1bad-4df6-b60d-46ed7b92f610-proxy-tls\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.268587 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.268561    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:40:53.274069 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.274050    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfkqg\" (UniqueName: \"kubernetes.io/projected/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-api-access-gfkqg\") pod \"raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.352431 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.352406    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:40:53.392568 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.392531    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:53.467081 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.466676    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-cabundle-cert\") pod \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") "
Sep 19 08:40:53.467081 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.466747    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gktv4\" (UniqueName: \"kubernetes.io/projected/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-api-access-gktv4\") pod \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") "
Sep 19 08:40:53.467081 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.466779    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-rbac-proxy-sar-config\") pod \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") "
Sep 19 08:40:53.467081 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.466808    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-proxy-tls\") pod \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") "
Sep 19 08:40:53.467858 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.467181    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" (UID: "ed93f295-4ccf-4c18-b7ae-55a25b55ebdd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:40:53.467858 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.467244    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kserve-provision-location\") pod \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\" (UID: \"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd\") "
Sep 19 08:40:53.467858 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.467691    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" (UID: "ed93f295-4ccf-4c18-b7ae-55a25b55ebdd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:40:53.467858 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.467709    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:40:53.468303 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.468075    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" (UID: "ed93f295-4ccf-4c18-b7ae-55a25b55ebdd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:40:53.469564 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.469537    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" (UID: "ed93f295-4ccf-4c18-b7ae-55a25b55ebdd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:40:53.469665 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.469641    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-api-access-gktv4" (OuterVolumeSpecName: "kube-api-access-gktv4") pod "ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" (UID: "ed93f295-4ccf-4c18-b7ae-55a25b55ebdd"). InnerVolumeSpecName "kube-api-access-gktv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:40:53.568581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.568541    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gktv4\" (UniqueName: \"kubernetes.io/projected/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-api-access-gktv4\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:40:53.568581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.568572    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:40:53.568581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.568584    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:40:53.568581 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.568593    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:40:53.668293 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.668248    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"]
Sep 19 08:40:53.670248 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:40:53.670224    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e7b273_1bad_4df6_b60d_46ed7b92f610.slice/crio-877cd57b747c2e1282842f97e22ce996edfb3309dc8634c0ea63d91ad2413fec WatchSource:0}: Error finding container 877cd57b747c2e1282842f97e22ce996edfb3309dc8634c0ea63d91ad2413fec: Status 404 returned error can't find the container with id 877cd57b747c2e1282842f97e22ce996edfb3309dc8634c0ea63d91ad2413fec
Sep 19 08:40:53.988225 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.988190    2559 generic.go:356] "Generic (PLEG): container finished" podID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerID="ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565" exitCode=0
Sep 19 08:40:53.988432 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.988277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" event={"ID":"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd","Type":"ContainerDied","Data":"ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565"}
Sep 19 08:40:53.988432 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.988303    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"
Sep 19 08:40:53.988432 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.988352    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8" event={"ID":"ed93f295-4ccf-4c18-b7ae-55a25b55ebdd","Type":"ContainerDied","Data":"4e40a4d694060a7862c864f8d9c179856a799f0136160373cb483f24b2b76a09"}
Sep 19 08:40:53.988432 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.988389    2559 scope.go:117] "RemoveContainer" containerID="3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc"
Sep 19 08:40:53.990037 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.990008    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" event={"ID":"e7e7b273-1bad-4df6-b60d-46ed7b92f610","Type":"ContainerStarted","Data":"d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179"}
Sep 19 08:40:53.990146 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.990069    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" event={"ID":"e7e7b273-1bad-4df6-b60d-46ed7b92f610","Type":"ContainerStarted","Data":"877cd57b747c2e1282842f97e22ce996edfb3309dc8634c0ea63d91ad2413fec"}
Sep 19 08:40:53.997223 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:53.997206    2559 scope.go:117] "RemoveContainer" containerID="ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565"
Sep 19 08:40:54.005708 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:54.005688    2559 scope.go:117] "RemoveContainer" containerID="d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7"
Sep 19 08:40:54.007070 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:54.007051    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"]
Sep 19 08:40:54.010270 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:54.010237    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-655f6-predictor-f56454574-585d8"]
Sep 19 08:40:54.014659 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:54.014643    2559 scope.go:117] "RemoveContainer" containerID="3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc"
Sep 19 08:40:54.014969 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:40:54.014948    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc\": container with ID starting with 3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc not found: ID does not exist" containerID="3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc"
Sep 19 08:40:54.015041 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:54.014980    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc"} err="failed to get container status \"3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc\": rpc error: code = NotFound desc = could not find container \"3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc\": container with ID starting with 3ff088ba18c6359d8619039eb53fa3b5ee9c6868618520635be9f2d8fe9a33fc not found: ID does not exist"
Sep 19 08:40:54.015041 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:54.014999    2559 scope.go:117] "RemoveContainer" containerID="ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565"
Sep 19 08:40:54.015309 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:40:54.015292    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565\": container with ID starting with ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565 not found: ID does not exist" containerID="ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565"
Sep 19 08:40:54.015361 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:54.015314    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565"} err="failed to get container status \"ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565\": rpc error: code = NotFound desc = could not find container \"ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565\": container with ID starting with ac69d4992246d4fea0eb95d480a5024a51cb8108c906d43d7baa7c24cda38565 not found: ID does not exist"
Sep 19 08:40:54.015361 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:54.015328    2559 scope.go:117] "RemoveContainer" containerID="d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7"
Sep 19 08:40:54.015583 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:40:54.015565    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7\": container with ID starting with d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7 not found: ID does not exist" containerID="d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7"
Sep 19 08:40:54.015631 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:54.015588    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7"} err="failed to get container status \"d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7\": rpc error: code = NotFound desc = could not find container \"d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7\": container with ID starting with d131f8ce4840872a53edbb89242d1de46152f3c4e02e41a81cae2972a14531b7 not found: ID does not exist"
Sep 19 08:40:55.938179 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:55.938147    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" path="/var/lib/kubelet/pods/ed93f295-4ccf-4c18-b7ae-55a25b55ebdd/volumes"
Sep 19 08:40:58.010105 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:58.010006    2559 generic.go:356] "Generic (PLEG): container finished" podID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerID="d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179" exitCode=0
Sep 19 08:40:58.010105 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:58.010082    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" event={"ID":"e7e7b273-1bad-4df6-b60d-46ed7b92f610","Type":"ContainerDied","Data":"d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179"}
Sep 19 08:40:59.017531 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:59.017496    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" event={"ID":"e7e7b273-1bad-4df6-b60d-46ed7b92f610","Type":"ContainerStarted","Data":"2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b"}
Sep 19 08:40:59.017921 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:59.017558    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" event={"ID":"e7e7b273-1bad-4df6-b60d-46ed7b92f610","Type":"ContainerStarted","Data":"242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028"}
Sep 19 08:40:59.017921 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:59.017744    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:40:59.040556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:40:59.040508    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podStartSLOduration=2.175870228 podStartE2EDuration="6.040490485s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.864620257s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:40:59.038818743 +0000 UTC m=+1211.449341743" watchObservedRunningTime="2026-09-19 08:40:59.040490485 +0000 UTC m=+1211.451013485"
Sep 19 08:41:00.022591 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:00.022550    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:41:00.023704 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:00.023678    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 08:41:01.027451 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:01.027406    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 08:41:06.031463 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:06.031433    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:41:06.032142 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:06.032116    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 08:41:16.032987 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:16.032936    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 08:41:26.032245 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:26.032201    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 08:41:36.032764 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:36.032676    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 08:41:38.550533 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:38.550492    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:41:43.684590 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.684555    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:41:43.732667 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.732629    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"]
Sep 19 08:41:43.733066 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.733048    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container"
Sep 19 08:41:43.733159 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.733069    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container"
Sep 19 08:41:43.733159 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.733092    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="storage-initializer"
Sep 19 08:41:43.733159 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.733102    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="storage-initializer"
Sep 19 08:41:43.733159 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.733119    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kube-rbac-proxy"
Sep 19 08:41:43.733159 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.733129    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kube-rbac-proxy"
Sep 19 08:41:43.733451 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.733211    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kube-rbac-proxy"
Sep 19 08:41:43.733451 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.733225    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed93f295-4ccf-4c18-b7ae-55a25b55ebdd" containerName="kserve-container"
Sep 19 08:41:43.736551 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.736530    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.738902 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.738874    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 19 08:41:43.739078 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.739059    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 19 08:41:43.756457 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.756432    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"]
Sep 19 08:41:43.769594 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.769565    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.769729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.769636    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72718a53-fcc4-41bb-a934-d69f1c41db06-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.769729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.769663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.769729 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.769722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.769877 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.769770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddv7q\" (UniqueName: \"kubernetes.io/projected/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-api-access-ddv7q\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.773513 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.773491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:41:43.818638 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.818602    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"]
Sep 19 08:41:43.822380 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.822350    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.824901 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.824874    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 19 08:41:43.839270 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.834393    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"]
Sep 19 08:41:43.870203 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.870161    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99b95866-0f6d-431a-8063-90a7c9a027b9-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.870203 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.870203    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.870491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.870224    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99b95866-0f6d-431a-8063-90a7c9a027b9-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.870491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.870299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.870491 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:41:43.870338    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 19 08:41:43.870491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.870337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.870491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.870389    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72718a53-fcc4-41bb-a934-d69f1c41db06-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.870491 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:41:43.870422    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls podName:72718a53-fcc4-41bb-a934-d69f1c41db06 nodeName:}" failed. No retries permitted until 2026-09-19 08:41:44.370400485 +0000 UTC m=+1256.780923476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls") pod "isvc-canary-create-v2-predictor-667f95d746-8sk2w" (UID: "72718a53-fcc4-41bb-a934-d69f1c41db06") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 19 08:41:43.870491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.870442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.870491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.870474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v898\" (UniqueName: \"kubernetes.io/projected/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-api-access-8v898\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.870964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.870509    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.870964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.870549    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddv7q\" (UniqueName: \"kubernetes.io/projected/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-api-access-ddv7q\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.870964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.870690    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72718a53-fcc4-41bb-a934-d69f1c41db06-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.871227 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.871204    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.871366 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.871346    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.879556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.879528    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddv7q\" (UniqueName: \"kubernetes.io/projected/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-api-access-ddv7q\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:43.971079 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.970982    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99b95866-0f6d-431a-8063-90a7c9a027b9-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.971079 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.971050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99b95866-0f6d-431a-8063-90a7c9a027b9-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.971377 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.971093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.971377 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.971115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.971377 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.971147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v898\" (UniqueName: \"kubernetes.io/projected/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-api-access-8v898\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.971377 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:41:43.971226    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 19 08:41:43.971377 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:41:43.971335    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99b95866-0f6d-431a-8063-90a7c9a027b9-proxy-tls podName:99b95866-0f6d-431a-8063-90a7c9a027b9 nodeName:}" failed. No retries permitted until 2026-09-19 08:41:44.471313501 +0000 UTC m=+1256.881836491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/99b95866-0f6d-431a-8063-90a7c9a027b9-proxy-tls") pod "isvc-canary-create-predictor-76b74f4d6c-2qkjw" (UID: "99b95866-0f6d-431a-8063-90a7c9a027b9") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 19 08:41:43.971633 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.971450    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99b95866-0f6d-431a-8063-90a7c9a027b9-kserve-provision-location\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.971775 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.971757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-cabundle-cert\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.971812 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.971782    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.981650 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.981632    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v898\" (UniqueName: \"kubernetes.io/projected/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-api-access-8v898\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:43.994372 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:43.994317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:41:44.182310 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:44.182249    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:41:44.374783 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:44.374741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:44.377286 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:44.377244    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls\") pod \"isvc-canary-create-v2-predictor-667f95d746-8sk2w\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:44.475302 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:44.475240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99b95866-0f6d-431a-8063-90a7c9a027b9-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:44.477795 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:44.477765    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99b95866-0f6d-431a-8063-90a7c9a027b9-proxy-tls\") pod \"isvc-canary-create-predictor-76b74f4d6c-2qkjw\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:44.648755 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:44.648653    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:44.734797 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:44.734769    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:44.925007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:44.924925    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"]
Sep 19 08:41:44.928379 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:41:44.928345    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72718a53_fcc4_41bb_a934_d69f1c41db06.slice/crio-3a7e801a58412f4118f95f556970511eab224d7d43c951920b141d1b0140c6ab WatchSource:0}: Error finding container 3a7e801a58412f4118f95f556970511eab224d7d43c951920b141d1b0140c6ab: Status 404 returned error can't find the container with id 3a7e801a58412f4118f95f556970511eab224d7d43c951920b141d1b0140c6ab
Sep 19 08:41:45.008450 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:45.008426    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"]
Sep 19 08:41:45.010732 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:41:45.010701    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b95866_0f6d_431a_8063_90a7c9a027b9.slice/crio-79cad0aeaf3e63ade9dd1087677f6b73af29d67d9b8725ad76a481c0cd5c7eb4 WatchSource:0}: Error finding container 79cad0aeaf3e63ade9dd1087677f6b73af29d67d9b8725ad76a481c0cd5c7eb4: Status 404 returned error can't find the container with id 79cad0aeaf3e63ade9dd1087677f6b73af29d67d9b8725ad76a481c0cd5c7eb4
Sep 19 08:41:45.234355 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:45.234219    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" event={"ID":"99b95866-0f6d-431a-8063-90a7c9a027b9","Type":"ContainerStarted","Data":"0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e"}
Sep 19 08:41:45.234355 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:45.234322    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" event={"ID":"99b95866-0f6d-431a-8063-90a7c9a027b9","Type":"ContainerStarted","Data":"79cad0aeaf3e63ade9dd1087677f6b73af29d67d9b8725ad76a481c0cd5c7eb4"}
Sep 19 08:41:45.236078 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:45.236050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" event={"ID":"72718a53-fcc4-41bb-a934-d69f1c41db06","Type":"ContainerStarted","Data":"e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1"}
Sep 19 08:41:45.236241 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:45.236109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" event={"ID":"72718a53-fcc4-41bb-a934-d69f1c41db06","Type":"ContainerStarted","Data":"3a7e801a58412f4118f95f556970511eab224d7d43c951920b141d1b0140c6ab"}
Sep 19 08:41:46.032105 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:46.032069    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 08:41:49.259187 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:49.259152    2559 generic.go:356] "Generic (PLEG): container finished" podID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerID="0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e" exitCode=0
Sep 19 08:41:49.259632 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:49.259231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" event={"ID":"99b95866-0f6d-431a-8063-90a7c9a027b9","Type":"ContainerDied","Data":"0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e"}
Sep 19 08:41:49.260992 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:49.260969    2559 generic.go:356] "Generic (PLEG): container finished" podID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerID="e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1" exitCode=0
Sep 19 08:41:49.261099 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:49.261027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" event={"ID":"72718a53-fcc4-41bb-a934-d69f1c41db06","Type":"ContainerDied","Data":"e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1"}
Sep 19 08:41:50.268297 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:50.268185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" event={"ID":"72718a53-fcc4-41bb-a934-d69f1c41db06","Type":"ContainerStarted","Data":"d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303"}
Sep 19 08:41:50.268297 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:50.268278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" event={"ID":"72718a53-fcc4-41bb-a934-d69f1c41db06","Type":"ContainerStarted","Data":"5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa"}
Sep 19 08:41:50.268773 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:50.268607    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:50.270305 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:50.270280    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" event={"ID":"99b95866-0f6d-431a-8063-90a7c9a027b9","Type":"ContainerStarted","Data":"e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc"}
Sep 19 08:41:50.270441 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:50.270333    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" event={"ID":"99b95866-0f6d-431a-8063-90a7c9a027b9","Type":"ContainerStarted","Data":"6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962"}
Sep 19 08:41:50.270529 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:50.270514    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:50.290829 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:50.290780    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podStartSLOduration=3.258056324 podStartE2EDuration="7.290764063s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.032707739s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:41:50.288171336 +0000 UTC m=+1262.698694349" watchObservedRunningTime="2026-09-19 08:41:50.290764063 +0000 UTC m=+1262.701287063"
Sep 19 08:41:50.309615 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:50.309570    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podStartSLOduration=3.329663627 podStartE2EDuration="7.309556074s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.979892447s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:41:50.30819582 +0000 UTC m=+1262.718718820" watchObservedRunningTime="2026-09-19 08:41:50.309556074 +0000 UTC m=+1262.720079073"
Sep 19 08:41:51.281704 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:51.281673    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:51.282097 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:51.281758    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:51.283012 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:51.282985    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 19 08:41:51.283126 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:51.283040    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 19 08:41:52.286186 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:52.286143    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 19 08:41:52.286186 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:52.286161    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 19 08:41:56.032432 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:56.032387    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 08:41:57.290411 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:57.290378    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:41:57.290968 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:57.290584    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:41:57.290968 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:57.290789    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 19 08:41:57.291366 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:41:57.291344    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 19 08:42:06.032692 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:06.032654    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 08:42:07.290930 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:07.290893    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 19 08:42:07.291447 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:07.291416    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 19 08:42:16.032220 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:16.032170    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 08:42:17.291671 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:17.291635    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 19 08:42:17.292058 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:17.291660    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 19 08:42:26.032418 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:26.032387    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:42:27.290801 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:27.290765    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 19 08:42:27.291274 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:27.291227    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 19 08:42:33.859139 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:33.859104    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"]
Sep 19 08:42:33.859697 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:33.859361    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" containerID="cri-o://242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028" gracePeriod=30
Sep 19 08:42:33.859697 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:33.859411    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kube-rbac-proxy" containerID="cri-o://2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b" gracePeriod=30
Sep 19 08:42:33.875797 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:33.875775    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:42:34.493083 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:34.493051    2559 generic.go:356] "Generic (PLEG): container finished" podID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerID="2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b" exitCode=2
Sep 19 08:42:34.493314 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:34.493121    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" event={"ID":"e7e7b273-1bad-4df6-b60d-46ed7b92f610","Type":"ContainerDied","Data":"2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b"}
Sep 19 08:42:36.027777 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:36.027734    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 19 08:42:36.032063 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:36.032033    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 19 08:42:37.291319 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:37.291285    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 19 08:42:37.291319 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:37.291304    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 19 08:42:39.406813 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.406789    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:42:39.483644 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.483609    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e7b273-1bad-4df6-b60d-46ed7b92f610-proxy-tls\") pod \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") "
Sep 19 08:42:39.483829 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.483704    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfkqg\" (UniqueName: \"kubernetes.io/projected/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-api-access-gfkqg\") pod \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") "
Sep 19 08:42:39.483829 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.483740    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-cabundle-cert\") pod \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") "
Sep 19 08:42:39.483829 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.483785    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kserve-provision-location\") pod \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") "
Sep 19 08:42:39.483998 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.483855    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-rbac-proxy-sar-config\") pod \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\" (UID: \"e7e7b273-1bad-4df6-b60d-46ed7b92f610\") "
Sep 19 08:42:39.484086 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.484058    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7e7b273-1bad-4df6-b60d-46ed7b92f610" (UID: "e7e7b273-1bad-4df6-b60d-46ed7b92f610"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:39.484163 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.484131    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7e7b273-1bad-4df6-b60d-46ed7b92f610" (UID: "e7e7b273-1bad-4df6-b60d-46ed7b92f610"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:42:39.484244 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.484224    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7e7b273-1bad-4df6-b60d-46ed7b92f610" (UID: "e7e7b273-1bad-4df6-b60d-46ed7b92f610"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:42:39.485763 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.485730    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e7b273-1bad-4df6-b60d-46ed7b92f610-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7e7b273-1bad-4df6-b60d-46ed7b92f610" (UID: "e7e7b273-1bad-4df6-b60d-46ed7b92f610"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:42:39.485878 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.485786    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-api-access-gfkqg" (OuterVolumeSpecName: "kube-api-access-gfkqg") pod "e7e7b273-1bad-4df6-b60d-46ed7b92f610" (UID: "e7e7b273-1bad-4df6-b60d-46ed7b92f610"). InnerVolumeSpecName "kube-api-access-gfkqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:42:39.518015 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.517936    2559 generic.go:356] "Generic (PLEG): container finished" podID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerID="242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028" exitCode=0
Sep 19 08:42:39.518015 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.517992    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"
Sep 19 08:42:39.518210 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.518018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" event={"ID":"e7e7b273-1bad-4df6-b60d-46ed7b92f610","Type":"ContainerDied","Data":"242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028"}
Sep 19 08:42:39.518210 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.518078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv" event={"ID":"e7e7b273-1bad-4df6-b60d-46ed7b92f610","Type":"ContainerDied","Data":"877cd57b747c2e1282842f97e22ce996edfb3309dc8634c0ea63d91ad2413fec"}
Sep 19 08:42:39.518210 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.518106    2559 scope.go:117] "RemoveContainer" containerID="2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b"
Sep 19 08:42:39.527027 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.527005    2559 scope.go:117] "RemoveContainer" containerID="242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028"
Sep 19 08:42:39.534775 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.534754    2559 scope.go:117] "RemoveContainer" containerID="d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179"
Sep 19 08:42:39.539752 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.539730    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"]
Sep 19 08:42:39.544371 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.544349    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-44db7-predictor-84b6797678-cd9wv"]
Sep 19 08:42:39.546694 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.546677    2559 scope.go:117] "RemoveContainer" containerID="2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b"
Sep 19 08:42:39.547091 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:42:39.547068    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b\": container with ID starting with 2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b not found: ID does not exist" containerID="2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b"
Sep 19 08:42:39.547167 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.547103    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b"} err="failed to get container status \"2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b\": rpc error: code = NotFound desc = could not find container \"2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b\": container with ID starting with 2879525f548e3e0df5a1445b93b0fa0816b1ea60851b8f6ee2393955f2e5920b not found: ID does not exist"
Sep 19 08:42:39.547167 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.547129    2559 scope.go:117] "RemoveContainer" containerID="242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028"
Sep 19 08:42:39.547495 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:42:39.547473    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028\": container with ID starting with 242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028 not found: ID does not exist" containerID="242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028"
Sep 19 08:42:39.547561 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.547506    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028"} err="failed to get container status \"242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028\": rpc error: code = NotFound desc = could not find container \"242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028\": container with ID starting with 242f509f8fce71cdfd6ce830ee83b47fe491cb0e996779096680fa6f0785b028 not found: ID does not exist"
Sep 19 08:42:39.547561 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.547530    2559 scope.go:117] "RemoveContainer" containerID="d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179"
Sep 19 08:42:39.547818 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:42:39.547802    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179\": container with ID starting with d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179 not found: ID does not exist" containerID="d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179"
Sep 19 08:42:39.547876 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.547822    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179"} err="failed to get container status \"d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179\": rpc error: code = NotFound desc = could not find container \"d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179\": container with ID starting with d5177e8023be28a55d806a5b7aaaad68c45742e20f4b0bd58df3d9715b2dc179 not found: ID does not exist"
Sep 19 08:42:39.584663 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.584635    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e7b273-1bad-4df6-b60d-46ed7b92f610-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:42:39.584663 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.584663    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfkqg\" (UniqueName: \"kubernetes.io/projected/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-api-access-gfkqg\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:42:39.584825 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.584672    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:42:39.584825 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.584680    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:42:39.584825 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.584689    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7e7b273-1bad-4df6-b60d-46ed7b92f610-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:42:39.938364 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:39.938332    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" path="/var/lib/kubelet/pods/e7e7b273-1bad-4df6-b60d-46ed7b92f610/volumes"
Sep 19 08:42:47.291308 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:47.291115    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 19 08:42:47.291308 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:47.291278    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 19 08:42:57.291467 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:57.291423    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 19 08:42:57.291986 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:42:57.291468    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 19 08:43:03.933420 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:03.933381    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 19 08:43:07.290764 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:07.290727    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 19 08:43:12.933467 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:12.933430    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:43:13.062888 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:13.062851    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"]
Sep 19 08:43:13.063121 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:13.063090    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container" containerID="cri-o://6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962" gracePeriod=30
Sep 19 08:43:13.063276 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:13.063158    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kube-rbac-proxy" containerID="cri-o://e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc" gracePeriod=30
Sep 19 08:43:13.064022 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:13.064002    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:43:13.687392 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:13.687358    2559 generic.go:356] "Generic (PLEG): container finished" podID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerID="e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc" exitCode=2
Sep 19 08:43:13.687574 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:13.687430    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" event={"ID":"99b95866-0f6d-431a-8063-90a7c9a027b9","Type":"ContainerDied","Data":"e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc"}
Sep 19 08:43:13.786121 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:13.785920    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 19 08:43:13.786121 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:13.786005    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls podName:72718a53-fcc4-41bb-a934-d69f1c41db06 nodeName:}" failed. No retries permitted until 2026-09-19 08:43:14.285981535 +0000 UTC m=+1346.696504536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls") pod "isvc-canary-create-v2-predictor-667f95d746-8sk2w" (UID: "72718a53-fcc4-41bb-a934-d69f1c41db06") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 19 08:43:13.797608 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:13.797573    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"]
Sep 19 08:43:13.804455 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:13.804418    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:43:13.863501 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:13.863467    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:43:14.290352 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:14.290317    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 19 08:43:14.290763 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:14.290403    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls podName:72718a53-fcc4-41bb-a934-d69f1c41db06 nodeName:}" failed. No retries permitted until 2026-09-19 08:43:15.290386718 +0000 UTC m=+1347.700909696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls") pod "isvc-canary-create-v2-predictor-667f95d746-8sk2w" (UID: "72718a53-fcc4-41bb-a934-d69f1c41db06") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 19 08:43:14.692419 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:14.692319    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container" containerID="cri-o://5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa" gracePeriod=30
Sep 19 08:43:14.692600 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:14.692330    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kube-rbac-proxy" containerID="cri-o://d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303" gracePeriod=30
Sep 19 08:43:15.299433 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:15.299399    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 19 08:43:15.299823 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:15.299483    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls podName:72718a53-fcc4-41bb-a934-d69f1c41db06 nodeName:}" failed. No retries permitted until 2026-09-19 08:43:17.299467837 +0000 UTC m=+1349.709990816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls") pod "isvc-canary-create-v2-predictor-667f95d746-8sk2w" (UID: "72718a53-fcc4-41bb-a934-d69f1c41db06") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 19 08:43:15.699519 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:15.699422    2559 generic.go:356] "Generic (PLEG): container finished" podID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerID="d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303" exitCode=2
Sep 19 08:43:15.699519 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:15.699502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" event={"ID":"72718a53-fcc4-41bb-a934-d69f1c41db06","Type":"ContainerDied","Data":"d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303"}
Sep 19 08:43:17.287227 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:17.287188    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 19 08:43:17.287671 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:17.287200    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 19 08:43:17.320515 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:17.320473    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 19 08:43:17.320715 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:17.320551    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls podName:72718a53-fcc4-41bb-a934-d69f1c41db06 nodeName:}" failed. No retries permitted until 2026-09-19 08:43:21.320533826 +0000 UTC m=+1353.731056804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls") pod "isvc-canary-create-v2-predictor-667f95d746-8sk2w" (UID: "72718a53-fcc4-41bb-a934-d69f1c41db06") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 19 08:43:18.513023 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.512997    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:43:18.632807 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.632712    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99b95866-0f6d-431a-8063-90a7c9a027b9-proxy-tls\") pod \"99b95866-0f6d-431a-8063-90a7c9a027b9\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") "
Sep 19 08:43:18.632807 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.632748    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-cabundle-cert\") pod \"99b95866-0f6d-431a-8063-90a7c9a027b9\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") "
Sep 19 08:43:18.632807 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.632808    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99b95866-0f6d-431a-8063-90a7c9a027b9-kserve-provision-location\") pod \"99b95866-0f6d-431a-8063-90a7c9a027b9\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") "
Sep 19 08:43:18.633108 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.632874    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v898\" (UniqueName: \"kubernetes.io/projected/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-api-access-8v898\") pod \"99b95866-0f6d-431a-8063-90a7c9a027b9\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") "
Sep 19 08:43:18.633108 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.632896    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-rbac-proxy-sar-config\") pod \"99b95866-0f6d-431a-8063-90a7c9a027b9\" (UID: \"99b95866-0f6d-431a-8063-90a7c9a027b9\") "
Sep 19 08:43:18.633249 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.633219    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b95866-0f6d-431a-8063-90a7c9a027b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99b95866-0f6d-431a-8063-90a7c9a027b9" (UID: "99b95866-0f6d-431a-8063-90a7c9a027b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:18.633249 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.633237    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99b95866-0f6d-431a-8063-90a7c9a027b9" (UID: "99b95866-0f6d-431a-8063-90a7c9a027b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:43:18.633394 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.633327    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99b95866-0f6d-431a-8063-90a7c9a027b9" (UID: "99b95866-0f6d-431a-8063-90a7c9a027b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:43:18.635009 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.634988    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b95866-0f6d-431a-8063-90a7c9a027b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99b95866-0f6d-431a-8063-90a7c9a027b9" (UID: "99b95866-0f6d-431a-8063-90a7c9a027b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:43:18.635098 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.635031    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-api-access-8v898" (OuterVolumeSpecName: "kube-api-access-8v898") pod "99b95866-0f6d-431a-8063-90a7c9a027b9" (UID: "99b95866-0f6d-431a-8063-90a7c9a027b9"). InnerVolumeSpecName "kube-api-access-8v898". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:43:18.716962 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.716926    2559 generic.go:356] "Generic (PLEG): container finished" podID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerID="6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962" exitCode=0
Sep 19 08:43:18.717161 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.716992    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"
Sep 19 08:43:18.717161 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.717007    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" event={"ID":"99b95866-0f6d-431a-8063-90a7c9a027b9","Type":"ContainerDied","Data":"6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962"}
Sep 19 08:43:18.717161 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.717086    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw" event={"ID":"99b95866-0f6d-431a-8063-90a7c9a027b9","Type":"ContainerDied","Data":"79cad0aeaf3e63ade9dd1087677f6b73af29d67d9b8725ad76a481c0cd5c7eb4"}
Sep 19 08:43:18.717161 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.717123    2559 scope.go:117] "RemoveContainer" containerID="e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc"
Sep 19 08:43:18.725914 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.725898    2559 scope.go:117] "RemoveContainer" containerID="6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962"
Sep 19 08:43:18.733537 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.733520    2559 scope.go:117] "RemoveContainer" containerID="0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e"
Sep 19 08:43:18.733988 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.733969    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99b95866-0f6d-431a-8063-90a7c9a027b9-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:43:18.734074 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.733992    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v898\" (UniqueName: \"kubernetes.io/projected/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-api-access-8v898\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:43:18.734074 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.734007    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:43:18.734074 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.734017    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99b95866-0f6d-431a-8063-90a7c9a027b9-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:43:18.734074 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.734028    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99b95866-0f6d-431a-8063-90a7c9a027b9-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:43:18.739517 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.739494    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"]
Sep 19 08:43:18.742093 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.742071    2559 scope.go:117] "RemoveContainer" containerID="e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc"
Sep 19 08:43:18.742426 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:18.742403    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc\": container with ID starting with e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc not found: ID does not exist" containerID="e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc"
Sep 19 08:43:18.742491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.742438    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc"} err="failed to get container status \"e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc\": rpc error: code = NotFound desc = could not find container \"e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc\": container with ID starting with e7b3060494568e14ad57bbd43980b9c03d8b14af95348e719d89ce495bb77dfc not found: ID does not exist"
Sep 19 08:43:18.742491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.742465    2559 scope.go:117] "RemoveContainer" containerID="6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962"
Sep 19 08:43:18.742755 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:18.742735    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962\": container with ID starting with 6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962 not found: ID does not exist" containerID="6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962"
Sep 19 08:43:18.742816 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.742759    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962"} err="failed to get container status \"6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962\": rpc error: code = NotFound desc = could not find container \"6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962\": container with ID starting with 6b7af62486ceb127feed2ec6a23d5faa7e1724fc9112c1b2d8a46fbfb3d9b962 not found: ID does not exist"
Sep 19 08:43:18.742816 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.742776    2559 scope.go:117] "RemoveContainer" containerID="0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e"
Sep 19 08:43:18.743027 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:18.743010    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e\": container with ID starting with 0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e not found: ID does not exist" containerID="0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e"
Sep 19 08:43:18.743069 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.743040    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e"} err="failed to get container status \"0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e\": rpc error: code = NotFound desc = could not find container \"0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e\": container with ID starting with 0f2fcbadc60187afde5d99453d534e41df1afc65eb76a808533a11056590765e not found: ID does not exist"
Sep 19 08:43:18.743774 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.743757    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-76b74f4d6c-2qkjw"]
Sep 19 08:43:18.990884 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:18.990739    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:43:19.060022 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.059986    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"]
Sep 19 08:43:19.060487 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060465    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kube-rbac-proxy"
Sep 19 08:43:19.060487 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060486    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kube-rbac-proxy"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060496    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="storage-initializer"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060504    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="storage-initializer"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060519    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060528    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060537    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kube-rbac-proxy"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060543    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kube-rbac-proxy"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060559    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="storage-initializer"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060567    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="storage-initializer"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060575    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060583    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060686    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kube-rbac-proxy"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060702    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e7b273-1bad-4df6-b60d-46ed7b92f610" containerName="kserve-container"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060711    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kube-rbac-proxy"
Sep 19 08:43:19.060749 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.060721    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" containerName="kserve-container"
Sep 19 08:43:19.065571 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.065552    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.068011 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.067990    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 19 08:43:19.068125 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.067990    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 19 08:43:19.069953 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.069933    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:43:19.076029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.076008    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"]
Sep 19 08:43:19.120990 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.120956    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"]
Sep 19 08:43:19.124543 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.124526    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.127098 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.127070    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 19 08:43:19.133574 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.133549    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"]
Sep 19 08:43:19.138099 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.138075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c76abe-887e-4ab7-8225-182a17fa6559-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.138196 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.138110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d599a84-d73d-480d-80f9-846dd09f8d9b-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.138196 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.138157    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llw2g\" (UniqueName: \"kubernetes.io/projected/12c76abe-887e-4ab7-8225-182a17fa6559-kube-api-access-llw2g\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.138196 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.138191    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.138377 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.138212    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12c76abe-887e-4ab7-8225-182a17fa6559-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.138377 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.138315    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8drd\" (UniqueName: \"kubernetes.io/projected/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-api-access-s8drd\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.138377 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.138357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.138538 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.138415    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.138538 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.138464    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.138538 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.138505    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d599a84-d73d-480d-80f9-846dd09f8d9b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.146423 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.146395    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:43:19.239763 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.239725    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c76abe-887e-4ab7-8225-182a17fa6559-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.239964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.239779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d599a84-d73d-480d-80f9-846dd09f8d9b-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.239964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.239810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llw2g\" (UniqueName: \"kubernetes.io/projected/12c76abe-887e-4ab7-8225-182a17fa6559-kube-api-access-llw2g\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.239964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.239841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.239964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.239862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12c76abe-887e-4ab7-8225-182a17fa6559-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.239964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.239885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8drd\" (UniqueName: \"kubernetes.io/projected/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-api-access-s8drd\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.239964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.239904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.239964 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.239950    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.239964 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:19.239955    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 19 08:43:19.240373 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.239996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.240373 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:19.240039    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d599a84-d73d-480d-80f9-846dd09f8d9b-proxy-tls podName:0d599a84-d73d-480d-80f9-846dd09f8d9b nodeName:}" failed. No retries permitted until 2026-09-19 08:43:19.74001771 +0000 UTC m=+1352.150540708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d599a84-d73d-480d-80f9-846dd09f8d9b-proxy-tls") pod "isvc-canary-promote-predictor-6546f8697f-k5m8f" (UID: "0d599a84-d73d-480d-80f9-846dd09f8d9b") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 19 08:43:19.240373 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.240067    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d599a84-d73d-480d-80f9-846dd09f8d9b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.240539 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.240469    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d599a84-d73d-480d-80f9-846dd09f8d9b-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.240844 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.240823    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.240967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.240853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-cabundle-cert\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.240967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.240889    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12c76abe-887e-4ab7-8225-182a17fa6559-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.241112 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.241086    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.241216 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.241142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.242503 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.242484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c76abe-887e-4ab7-8225-182a17fa6559-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.248793 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.248755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8drd\" (UniqueName: \"kubernetes.io/projected/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-api-access-s8drd\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.248793 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.248767    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llw2g\" (UniqueName: \"kubernetes.io/projected/12c76abe-887e-4ab7-8225-182a17fa6559-kube-api-access-llw2g\") pod \"isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.377154 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.377114    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:19.653934 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.653907    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:43:19.654285 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.654217    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"]
Sep 19 08:43:19.656742 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:43:19.656715    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c76abe_887e_4ab7_8225_182a17fa6559.slice/crio-4f881bd6fea3d59195b6c7d2afccaf5f27974bd5b9845638db5eca29e70df6c0 WatchSource:0}: Error finding container 4f881bd6fea3d59195b6c7d2afccaf5f27974bd5b9845638db5eca29e70df6c0: Status 404 returned error can't find the container with id 4f881bd6fea3d59195b6c7d2afccaf5f27974bd5b9845638db5eca29e70df6c0
Sep 19 08:43:19.725648 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.725619    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" event={"ID":"12c76abe-887e-4ab7-8225-182a17fa6559","Type":"ContainerStarted","Data":"4f881bd6fea3d59195b6c7d2afccaf5f27974bd5b9845638db5eca29e70df6c0"}
Sep 19 08:43:19.744703 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.744677    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d599a84-d73d-480d-80f9-846dd09f8d9b-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.746917 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.746890    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d599a84-d73d-480d-80f9-846dd09f8d9b-proxy-tls\") pod \"isvc-canary-promote-predictor-6546f8697f-k5m8f\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:19.939176 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:19.939088    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b95866-0f6d-431a-8063-90a7c9a027b9" path="/var/lib/kubelet/pods/99b95866-0f6d-431a-8063-90a7c9a027b9/volumes"
Sep 19 08:43:20.035623 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.035587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:20.314065 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.314032    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"]
Sep 19 08:43:20.315423 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:43:20.315390    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d599a84_d73d_480d_80f9_846dd09f8d9b.slice/crio-03bc053a7e2517320517b42f14b0354bcde2e04119e72bd882d1b6799dcd0b67 WatchSource:0}: Error finding container 03bc053a7e2517320517b42f14b0354bcde2e04119e72bd882d1b6799dcd0b67: Status 404 returned error can't find the container with id 03bc053a7e2517320517b42f14b0354bcde2e04119e72bd882d1b6799dcd0b67
Sep 19 08:43:20.345349 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.345325    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:43:20.452226 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.452189    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-cabundle-cert\") pod \"72718a53-fcc4-41bb-a934-d69f1c41db06\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") "
Sep 19 08:43:20.452436 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.452289    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddv7q\" (UniqueName: \"kubernetes.io/projected/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-api-access-ddv7q\") pod \"72718a53-fcc4-41bb-a934-d69f1c41db06\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") "
Sep 19 08:43:20.452436 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.452331    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls\") pod \"72718a53-fcc4-41bb-a934-d69f1c41db06\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") "
Sep 19 08:43:20.452436 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.452394    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72718a53-fcc4-41bb-a934-d69f1c41db06-kserve-provision-location\") pod \"72718a53-fcc4-41bb-a934-d69f1c41db06\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") "
Sep 19 08:43:20.452436 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.452417    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-rbac-proxy-sar-config\") pod \"72718a53-fcc4-41bb-a934-d69f1c41db06\" (UID: \"72718a53-fcc4-41bb-a934-d69f1c41db06\") "
Sep 19 08:43:20.452656 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.452576    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "72718a53-fcc4-41bb-a934-d69f1c41db06" (UID: "72718a53-fcc4-41bb-a934-d69f1c41db06"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:43:20.452757 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.452729    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72718a53-fcc4-41bb-a934-d69f1c41db06-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "72718a53-fcc4-41bb-a934-d69f1c41db06" (UID: "72718a53-fcc4-41bb-a934-d69f1c41db06"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:20.452866 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.452840    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "72718a53-fcc4-41bb-a934-d69f1c41db06" (UID: "72718a53-fcc4-41bb-a934-d69f1c41db06"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:43:20.454500 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.454473    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "72718a53-fcc4-41bb-a934-d69f1c41db06" (UID: "72718a53-fcc4-41bb-a934-d69f1c41db06"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:43:20.454631 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.454606    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-api-access-ddv7q" (OuterVolumeSpecName: "kube-api-access-ddv7q") pod "72718a53-fcc4-41bb-a934-d69f1c41db06" (UID: "72718a53-fcc4-41bb-a934-d69f1c41db06"). InnerVolumeSpecName "kube-api-access-ddv7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:43:20.553711 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.553675    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/72718a53-fcc4-41bb-a934-d69f1c41db06-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:43:20.553711 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.553704    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:43:20.553711 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.553717    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/72718a53-fcc4-41bb-a934-d69f1c41db06-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:43:20.553944 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.553727    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddv7q\" (UniqueName: \"kubernetes.io/projected/72718a53-fcc4-41bb-a934-d69f1c41db06-kube-api-access-ddv7q\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:43:20.553944 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.553736    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/72718a53-fcc4-41bb-a934-d69f1c41db06-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:43:20.733774 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.733694    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" event={"ID":"0d599a84-d73d-480d-80f9-846dd09f8d9b","Type":"ContainerStarted","Data":"cce990bc6c14b2673f3554a815a80a5eeee2d104b2b7bfadaaf8a8cb24a76a23"}
Sep 19 08:43:20.733774 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.733764    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" event={"ID":"0d599a84-d73d-480d-80f9-846dd09f8d9b","Type":"ContainerStarted","Data":"03bc053a7e2517320517b42f14b0354bcde2e04119e72bd882d1b6799dcd0b67"}
Sep 19 08:43:20.735935 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.735911    2559 generic.go:356] "Generic (PLEG): container finished" podID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerID="5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa" exitCode=0
Sep 19 08:43:20.736051 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.735971    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" event={"ID":"72718a53-fcc4-41bb-a934-d69f1c41db06","Type":"ContainerDied","Data":"5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa"}
Sep 19 08:43:20.736051 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.735983    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"
Sep 19 08:43:20.736051 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.736034    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w" event={"ID":"72718a53-fcc4-41bb-a934-d69f1c41db06","Type":"ContainerDied","Data":"3a7e801a58412f4118f95f556970511eab224d7d43c951920b141d1b0140c6ab"}
Sep 19 08:43:20.736222 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.736071    2559 scope.go:117] "RemoveContainer" containerID="d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303"
Sep 19 08:43:20.737941 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.737918    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" event={"ID":"12c76abe-887e-4ab7-8225-182a17fa6559","Type":"ContainerStarted","Data":"8d694d13fee068f0e5e43600d9a4ac64a1705de36daa4ef8f0491ebb5ea8fa98"}
Sep 19 08:43:20.745169 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.745152    2559 scope.go:117] "RemoveContainer" containerID="5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa"
Sep 19 08:43:20.753943 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.753925    2559 scope.go:117] "RemoveContainer" containerID="e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1"
Sep 19 08:43:20.762208 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.762184    2559 scope.go:117] "RemoveContainer" containerID="d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303"
Sep 19 08:43:20.762578 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:20.762559    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303\": container with ID starting with d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303 not found: ID does not exist" containerID="d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303"
Sep 19 08:43:20.762667 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.762590    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303"} err="failed to get container status \"d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303\": rpc error: code = NotFound desc = could not find container \"d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303\": container with ID starting with d81a8319756108ca351ae2d1b8ac08558043387f0efbbf66b38aa0c418fbb303 not found: ID does not exist"
Sep 19 08:43:20.762667 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.762615    2559 scope.go:117] "RemoveContainer" containerID="5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa"
Sep 19 08:43:20.762923 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:20.762900    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa\": container with ID starting with 5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa not found: ID does not exist" containerID="5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa"
Sep 19 08:43:20.762967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.762934    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa"} err="failed to get container status \"5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa\": rpc error: code = NotFound desc = could not find container \"5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa\": container with ID starting with 5c0a5244495b2980487f50c0cd44577242ad0d357f1ba594823f15958b5bf6aa not found: ID does not exist"
Sep 19 08:43:20.762967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.762963    2559 scope.go:117] "RemoveContainer" containerID="e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1"
Sep 19 08:43:20.763209 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:43:20.763191    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1\": container with ID starting with e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1 not found: ID does not exist" containerID="e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1"
Sep 19 08:43:20.763287 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.763215    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1"} err="failed to get container status \"e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1\": rpc error: code = NotFound desc = could not find container \"e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1\": container with ID starting with e9e2bde5c7229c0f038508ff0f1ab2ce5e4ed51c19fce2e8a22dd8bf407fa5a1 not found: ID does not exist"
Sep 19 08:43:20.766536 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.766516    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"]
Sep 19 08:43:20.769711 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:20.769691    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-667f95d746-8sk2w"]
Sep 19 08:43:21.938667 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:21.938632    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" path="/var/lib/kubelet/pods/72718a53-fcc4-41bb-a934-d69f1c41db06/volumes"
Sep 19 08:43:24.761578 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:24.761488    2559 generic.go:356] "Generic (PLEG): container finished" podID="12c76abe-887e-4ab7-8225-182a17fa6559" containerID="8d694d13fee068f0e5e43600d9a4ac64a1705de36daa4ef8f0491ebb5ea8fa98" exitCode=0
Sep 19 08:43:24.761578 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:24.761560    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" event={"ID":"12c76abe-887e-4ab7-8225-182a17fa6559","Type":"ContainerDied","Data":"8d694d13fee068f0e5e43600d9a4ac64a1705de36daa4ef8f0491ebb5ea8fa98"}
Sep 19 08:43:24.763166 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:24.763146    2559 generic.go:356] "Generic (PLEG): container finished" podID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerID="cce990bc6c14b2673f3554a815a80a5eeee2d104b2b7bfadaaf8a8cb24a76a23" exitCode=0
Sep 19 08:43:24.763267 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:24.763214    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" event={"ID":"0d599a84-d73d-480d-80f9-846dd09f8d9b","Type":"ContainerDied","Data":"cce990bc6c14b2673f3554a815a80a5eeee2d104b2b7bfadaaf8a8cb24a76a23"}
Sep 19 08:43:25.773268 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:25.773229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" event={"ID":"0d599a84-d73d-480d-80f9-846dd09f8d9b","Type":"ContainerStarted","Data":"009b7ddbcb1a9134da40512914c72c7b44b634e5d5c89356dcd62f801d2e2162"}
Sep 19 08:43:25.773759 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:25.773319    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" event={"ID":"0d599a84-d73d-480d-80f9-846dd09f8d9b","Type":"ContainerStarted","Data":"275a31fe2d4b767615a452df2707f76648e4237ee324a67f33feb1f58083cb5c"}
Sep 19 08:43:25.773759 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:25.773698    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:25.775414 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:25.775383    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" event={"ID":"12c76abe-887e-4ab7-8225-182a17fa6559","Type":"ContainerStarted","Data":"71195f482b85faf0e042009023af35157515f7016eebf40146a4fbb8757c95e2"}
Sep 19 08:43:25.775551 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:25.775443    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" event={"ID":"12c76abe-887e-4ab7-8225-182a17fa6559","Type":"ContainerStarted","Data":"4fc7abbaa02b12a62bcdf8cb9fa5983e4ff4f31c1c23cb7d327acd9b3a840738"}
Sep 19 08:43:25.775660 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:25.775643    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:25.793525 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:25.793481    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podStartSLOduration=2.820337688 podStartE2EDuration="6.793463955s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.973126267s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:25.791550109 +0000 UTC m=+1358.202073109" watchObservedRunningTime="2026-09-19 08:43:25.793463955 +0000 UTC m=+1358.203987008"
Sep 19 08:43:25.809892 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:25.809844    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podStartSLOduration=2.776443478 podStartE2EDuration="6.809829591s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.033386113s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:25.808841153 +0000 UTC m=+1358.219364155" watchObservedRunningTime="2026-09-19 08:43:25.809829591 +0000 UTC m=+1358.220352590"
Sep 19 08:43:26.780811 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:26.780774    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:26.781220 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:26.780835    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:26.781888 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:26.781858    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 19 08:43:26.782007 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:26.781970    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 08:43:27.786329 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:27.786289    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 08:43:27.786804 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:27.786380    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 19 08:43:32.791483 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:32.791454    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:43:32.791877 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:32.791541    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:43:32.792160 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:32.792125    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 19 08:43:32.792303 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:32.792153    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 08:43:42.792445 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:42.792407    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 19 08:43:42.792937 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:42.792409    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 08:43:52.792306 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:52.792234    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 08:43:52.792780 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:43:52.792234    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 19 08:44:02.792583 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:02.792545    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 08:44:02.793020 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:02.792542    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 19 08:44:12.792272 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:12.792212    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 08:44:12.792658 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:12.792225    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 19 08:44:22.792464 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:22.792421    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 08:44:22.793026 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:22.792423    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 19 08:44:32.792546 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:32.792459    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 08:44:32.792932 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:32.792456    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 19 08:44:37.933002 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:37.932951    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 19 08:44:42.793014 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:42.792972    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 19 08:44:47.938429 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:47.938399    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:44:52.793018 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:52.792988    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:44:52.925377 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:52.925339    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"]
Sep 19 08:44:52.925671 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:52.925640    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kube-rbac-proxy" containerID="cri-o://009b7ddbcb1a9134da40512914c72c7b44b634e5d5c89356dcd62f801d2e2162" gracePeriod=30
Sep 19 08:44:52.925774 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:52.925631    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" containerID="cri-o://275a31fe2d4b767615a452df2707f76648e4237ee324a67f33feb1f58083cb5c" gracePeriod=30
Sep 19 08:44:53.187395 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:53.187298    2559 generic.go:356] "Generic (PLEG): container finished" podID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerID="009b7ddbcb1a9134da40512914c72c7b44b634e5d5c89356dcd62f801d2e2162" exitCode=2
Sep 19 08:44:53.187395 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:53.187339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" event={"ID":"0d599a84-d73d-480d-80f9-846dd09f8d9b","Type":"ContainerDied","Data":"009b7ddbcb1a9134da40512914c72c7b44b634e5d5c89356dcd62f801d2e2162"}
Sep 19 08:44:54.144933 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:54.144888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:44:57.787337 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:57.787248    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.48:8643/healthz\": dial tcp 10.133.0.48:8643: connect: connection refused"
Sep 19 08:44:57.933031 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:57.932990    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 19 08:44:58.216202 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.216166    2559 generic.go:356] "Generic (PLEG): container finished" podID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerID="275a31fe2d4b767615a452df2707f76648e4237ee324a67f33feb1f58083cb5c" exitCode=0
Sep 19 08:44:58.216387 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.216235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" event={"ID":"0d599a84-d73d-480d-80f9-846dd09f8d9b","Type":"ContainerDied","Data":"275a31fe2d4b767615a452df2707f76648e4237ee324a67f33feb1f58083cb5c"}
Sep 19 08:44:58.278769 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.278746    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:44:58.331052 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.330972    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-cabundle-cert\") pod \"0d599a84-d73d-480d-80f9-846dd09f8d9b\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") "
Sep 19 08:44:58.331052 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.331047    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8drd\" (UniqueName: \"kubernetes.io/projected/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-api-access-s8drd\") pod \"0d599a84-d73d-480d-80f9-846dd09f8d9b\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") "
Sep 19 08:44:58.331241 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.331072    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-rbac-proxy-sar-config\") pod \"0d599a84-d73d-480d-80f9-846dd09f8d9b\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") "
Sep 19 08:44:58.331241 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.331129    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d599a84-d73d-480d-80f9-846dd09f8d9b-kserve-provision-location\") pod \"0d599a84-d73d-480d-80f9-846dd09f8d9b\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") "
Sep 19 08:44:58.331241 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.331157    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d599a84-d73d-480d-80f9-846dd09f8d9b-proxy-tls\") pod \"0d599a84-d73d-480d-80f9-846dd09f8d9b\" (UID: \"0d599a84-d73d-480d-80f9-846dd09f8d9b\") "
Sep 19 08:44:58.331506 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.331481    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d599a84-d73d-480d-80f9-846dd09f8d9b" (UID: "0d599a84-d73d-480d-80f9-846dd09f8d9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:44:58.331551 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.331486    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d599a84-d73d-480d-80f9-846dd09f8d9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d599a84-d73d-480d-80f9-846dd09f8d9b" (UID: "0d599a84-d73d-480d-80f9-846dd09f8d9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:44:58.331631 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.331609    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d599a84-d73d-480d-80f9-846dd09f8d9b" (UID: "0d599a84-d73d-480d-80f9-846dd09f8d9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:44:58.333297 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.333271    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-api-access-s8drd" (OuterVolumeSpecName: "kube-api-access-s8drd") pod "0d599a84-d73d-480d-80f9-846dd09f8d9b" (UID: "0d599a84-d73d-480d-80f9-846dd09f8d9b"). InnerVolumeSpecName "kube-api-access-s8drd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:44:58.333297 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.333283    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d599a84-d73d-480d-80f9-846dd09f8d9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d599a84-d73d-480d-80f9-846dd09f8d9b" (UID: "0d599a84-d73d-480d-80f9-846dd09f8d9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:44:58.432614 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.432558    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8drd\" (UniqueName: \"kubernetes.io/projected/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-api-access-s8drd\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:44:58.432614 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.432606    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:44:58.432614 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.432621    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d599a84-d73d-480d-80f9-846dd09f8d9b-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:44:58.432869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.432634    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d599a84-d73d-480d-80f9-846dd09f8d9b-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:44:58.432869 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:58.432647    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d599a84-d73d-480d-80f9-846dd09f8d9b-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:44:59.222988 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:59.222956    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"
Sep 19 08:44:59.223455 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:59.222956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f" event={"ID":"0d599a84-d73d-480d-80f9-846dd09f8d9b","Type":"ContainerDied","Data":"03bc053a7e2517320517b42f14b0354bcde2e04119e72bd882d1b6799dcd0b67"}
Sep 19 08:44:59.223455 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:59.223095    2559 scope.go:117] "RemoveContainer" containerID="009b7ddbcb1a9134da40512914c72c7b44b634e5d5c89356dcd62f801d2e2162"
Sep 19 08:44:59.232278 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:59.232245    2559 scope.go:117] "RemoveContainer" containerID="275a31fe2d4b767615a452df2707f76648e4237ee324a67f33feb1f58083cb5c"
Sep 19 08:44:59.239907 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:59.239888    2559 scope.go:117] "RemoveContainer" containerID="cce990bc6c14b2673f3554a815a80a5eeee2d104b2b7bfadaaf8a8cb24a76a23"
Sep 19 08:44:59.246088 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:59.246062    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"]
Sep 19 08:44:59.249691 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:59.249668    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6546f8697f-k5m8f"]
Sep 19 08:44:59.938166 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:44:59.938134    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" path="/var/lib/kubelet/pods/0d599a84-d73d-480d-80f9-846dd09f8d9b/volumes"
Sep 19 08:46:24.282635 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:24.282595    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"]
Sep 19 08:46:24.283248 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:24.282978    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container" containerID="cri-o://4fc7abbaa02b12a62bcdf8cb9fa5983e4ff4f31c1c23cb7d327acd9b3a840738" gracePeriod=30
Sep 19 08:46:24.283248 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:24.283097    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kube-rbac-proxy" containerID="cri-o://71195f482b85faf0e042009023af35157515f7016eebf40146a4fbb8757c95e2" gracePeriod=30
Sep 19 08:46:24.607481 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:24.607382    2559 generic.go:356] "Generic (PLEG): container finished" podID="12c76abe-887e-4ab7-8225-182a17fa6559" containerID="71195f482b85faf0e042009023af35157515f7016eebf40146a4fbb8757c95e2" exitCode=2
Sep 19 08:46:24.607481 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:24.607453    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" event={"ID":"12c76abe-887e-4ab7-8225-182a17fa6559","Type":"ContainerDied","Data":"71195f482b85faf0e042009023af35157515f7016eebf40146a4fbb8757c95e2"}
Sep 19 08:46:25.208136 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:25.208092    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:46:27.786648 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:27.786609    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 19 08:46:29.631940 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.631905    2559 generic.go:356] "Generic (PLEG): container finished" podID="12c76abe-887e-4ab7-8225-182a17fa6559" containerID="4fc7abbaa02b12a62bcdf8cb9fa5983e4ff4f31c1c23cb7d327acd9b3a840738" exitCode=0
Sep 19 08:46:29.632441 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.631972    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" event={"ID":"12c76abe-887e-4ab7-8225-182a17fa6559","Type":"ContainerDied","Data":"4fc7abbaa02b12a62bcdf8cb9fa5983e4ff4f31c1c23cb7d327acd9b3a840738"}
Sep 19 08:46:29.735311 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.735281    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:46:29.735573 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.735553    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c76abe-887e-4ab7-8225-182a17fa6559-proxy-tls\") pod \"12c76abe-887e-4ab7-8225-182a17fa6559\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") "
Sep 19 08:46:29.735632 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.735617    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12c76abe-887e-4ab7-8225-182a17fa6559-kserve-provision-location\") pod \"12c76abe-887e-4ab7-8225-182a17fa6559\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") "
Sep 19 08:46:29.735670 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.735658    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llw2g\" (UniqueName: \"kubernetes.io/projected/12c76abe-887e-4ab7-8225-182a17fa6559-kube-api-access-llw2g\") pod \"12c76abe-887e-4ab7-8225-182a17fa6559\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") "
Sep 19 08:46:29.735714 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.735697    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-kube-rbac-proxy-sar-config\") pod \"12c76abe-887e-4ab7-8225-182a17fa6559\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") "
Sep 19 08:46:29.735753 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.735736    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-cabundle-cert\") pod \"12c76abe-887e-4ab7-8225-182a17fa6559\" (UID: \"12c76abe-887e-4ab7-8225-182a17fa6559\") "
Sep 19 08:46:29.735900 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.735876    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12c76abe-887e-4ab7-8225-182a17fa6559-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "12c76abe-887e-4ab7-8225-182a17fa6559" (UID: "12c76abe-887e-4ab7-8225-182a17fa6559"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:29.736050 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.736032    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/12c76abe-887e-4ab7-8225-182a17fa6559-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:46:29.736131 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.736107    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12c76abe-887e-4ab7-8225-182a17fa6559" (UID: "12c76abe-887e-4ab7-8225-182a17fa6559"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:46:29.736193 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.736144    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "12c76abe-887e-4ab7-8225-182a17fa6559" (UID: "12c76abe-887e-4ab7-8225-182a17fa6559"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:46:29.737577 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.737555    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12c76abe-887e-4ab7-8225-182a17fa6559-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12c76abe-887e-4ab7-8225-182a17fa6559" (UID: "12c76abe-887e-4ab7-8225-182a17fa6559"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:46:29.737803 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.737786    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c76abe-887e-4ab7-8225-182a17fa6559-kube-api-access-llw2g" (OuterVolumeSpecName: "kube-api-access-llw2g") pod "12c76abe-887e-4ab7-8225-182a17fa6559" (UID: "12c76abe-887e-4ab7-8225-182a17fa6559"). InnerVolumeSpecName "kube-api-access-llw2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:46:29.836957 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.836871    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12c76abe-887e-4ab7-8225-182a17fa6559-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:46:29.836957 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.836901    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llw2g\" (UniqueName: \"kubernetes.io/projected/12c76abe-887e-4ab7-8225-182a17fa6559-kube-api-access-llw2g\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:46:29.836957 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.836910    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:46:29.836957 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:29.836919    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/12c76abe-887e-4ab7-8225-182a17fa6559-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:46:30.638441 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:30.638396    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn" event={"ID":"12c76abe-887e-4ab7-8225-182a17fa6559","Type":"ContainerDied","Data":"4f881bd6fea3d59195b6c7d2afccaf5f27974bd5b9845638db5eca29e70df6c0"}
Sep 19 08:46:30.638954 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:30.638461    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"
Sep 19 08:46:30.638954 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:30.638510    2559 scope.go:117] "RemoveContainer" containerID="71195f482b85faf0e042009023af35157515f7016eebf40146a4fbb8757c95e2"
Sep 19 08:46:30.647390 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:30.647364    2559 scope.go:117] "RemoveContainer" containerID="4fc7abbaa02b12a62bcdf8cb9fa5983e4ff4f31c1c23cb7d327acd9b3a840738"
Sep 19 08:46:30.654754 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:30.654737    2559 scope.go:117] "RemoveContainer" containerID="8d694d13fee068f0e5e43600d9a4ac64a1705de36daa4ef8f0491ebb5ea8fa98"
Sep 19 08:46:30.658080 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:30.658054    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"]
Sep 19 08:46:30.661588 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:30.661564    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-84fcdd9b5b-scxnn"]
Sep 19 08:46:30.670680 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:30.670656    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:46:31.938039 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:31.937995    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" path="/var/lib/kubelet/pods/12c76abe-887e-4ab7-8225-182a17fa6559/volumes"
Sep 19 08:46:38.733506 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.733467    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:46:38.792917 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.792885    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"]
Sep 19 08:46:38.793235 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793223    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="storage-initializer"
Sep 19 08:46:38.793307 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793236    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="storage-initializer"
Sep 19 08:46:38.793307 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793245    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container"
Sep 19 08:46:38.793307 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793272    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container"
Sep 19 08:46:38.793307 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793282    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kube-rbac-proxy"
Sep 19 08:46:38.793307 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793288    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kube-rbac-proxy"
Sep 19 08:46:38.793307 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793297    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container"
Sep 19 08:46:38.793307 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793302    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793312    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kube-rbac-proxy"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793317    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kube-rbac-proxy"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793325    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="storage-initializer"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793330    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="storage-initializer"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793338    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="storage-initializer"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793343    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="storage-initializer"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793349    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793354    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793361    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kube-rbac-proxy"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793366    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kube-rbac-proxy"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793420    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kube-rbac-proxy"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793427    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kserve-container"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793433    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kserve-container"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793441    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d599a84-d73d-480d-80f9-846dd09f8d9b" containerName="kserve-container"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793447    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="72718a53-fcc4-41bb-a934-d69f1c41db06" containerName="kube-rbac-proxy"
Sep 19 08:46:38.793526 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.793453    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="12c76abe-887e-4ab7-8225-182a17fa6559" containerName="kube-rbac-proxy"
Sep 19 08:46:38.796486 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.796470    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:38.799186 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.799156    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 19 08:46:38.799354 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.799196    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-b4prd\""
Sep 19 08:46:38.799354 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.799218    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 19 08:46:38.799354 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.799227    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 19 08:46:38.800000 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.799981    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 19 08:46:38.805676 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.805649    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"]
Sep 19 08:46:38.917659 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.917625    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r2r7\" (UniqueName: \"kubernetes.io/projected/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-api-access-6r2r7\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-brn8g\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:38.917851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.917668    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-brn8g\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:38.917851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:38.917699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa543ce-d637-4353-a4b3-c86bec90a8e3-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-brn8g\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:39.019288 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:39.019160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r2r7\" (UniqueName: \"kubernetes.io/projected/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-api-access-6r2r7\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-brn8g\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:39.019288 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:39.019209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-brn8g\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:39.019288 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:39.019235    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa543ce-d637-4353-a4b3-c86bec90a8e3-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-brn8g\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:39.019586 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:46:39.019434    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-serving-cert: secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 19 08:46:39.019586 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:46:39.019529    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aa543ce-d637-4353-a4b3-c86bec90a8e3-proxy-tls podName:3aa543ce-d637-4353-a4b3-c86bec90a8e3 nodeName:}" failed. No retries permitted until 2026-09-19 08:46:39.519508314 +0000 UTC m=+1551.930031306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3aa543ce-d637-4353-a4b3-c86bec90a8e3-proxy-tls") pod "message-dumper-raw-cipn-predictor-9774bf55d-brn8g" (UID: "3aa543ce-d637-4353-a4b3-c86bec90a8e3") : secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 19 08:46:39.019847 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:39.019829    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-brn8g\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:39.029885 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:39.029854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r2r7\" (UniqueName: \"kubernetes.io/projected/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-api-access-6r2r7\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-brn8g\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:39.230156 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:39.230120    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:46:39.524292 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:39.524238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa543ce-d637-4353-a4b3-c86bec90a8e3-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-brn8g\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:39.526748 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:39.526720    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa543ce-d637-4353-a4b3-c86bec90a8e3-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-brn8g\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:39.708228 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:39.708182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:39.980071 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:39.980049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"]
Sep 19 08:46:39.982180 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:46:39.982153    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa543ce_d637_4353_a4b3_c86bec90a8e3.slice/crio-3fc1c790d1790e6f09e8ceb0a64ef32f46699c76d5f42b4ff96b382a59387013 WatchSource:0}: Error finding container 3fc1c790d1790e6f09e8ceb0a64ef32f46699c76d5f42b4ff96b382a59387013: Status 404 returned error can't find the container with id 3fc1c790d1790e6f09e8ceb0a64ef32f46699c76d5f42b4ff96b382a59387013
Sep 19 08:46:40.687463 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:40.687421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g" event={"ID":"3aa543ce-d637-4353-a4b3-c86bec90a8e3","Type":"ContainerStarted","Data":"db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c"}
Sep 19 08:46:40.687668 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:40.687490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g" event={"ID":"3aa543ce-d637-4353-a4b3-c86bec90a8e3","Type":"ContainerStarted","Data":"14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3"}
Sep 19 08:46:40.687668 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:40.687525    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:40.687668 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:40.687547    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g" event={"ID":"3aa543ce-d637-4353-a4b3-c86bec90a8e3","Type":"ContainerStarted","Data":"3fc1c790d1790e6f09e8ceb0a64ef32f46699c76d5f42b4ff96b382a59387013"}
Sep 19 08:46:40.687668 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:40.687568    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:40.689412 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:40.689387    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:40.707004 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:40.706951    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g" podStartSLOduration=2.706936223 podStartE2EDuration="2.706936223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:46:40.704418192 +0000 UTC m=+1553.114941249" watchObservedRunningTime="2026-09-19 08:46:40.706936223 +0000 UTC m=+1553.117459223"
Sep 19 08:46:47.701703 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:47.701671    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:46:48.737540 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.737504    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:46:48.773662 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.773631    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"]
Sep 19 08:46:48.777624 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.777602    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:48.780422 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.780402    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 19 08:46:48.780553 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.780435    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 19 08:46:48.781039 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.781019    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 19 08:46:48.781117 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.781026    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:46:48.788616 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.788592    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"]
Sep 19 08:46:48.907895 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.907856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:48.907895 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.907899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:48.908118 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.907929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnqb5\" (UniqueName: \"kubernetes.io/projected/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-api-access-jnqb5\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:48.908118 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.907987    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:48.908118 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.908025    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:48.969213 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:48.969179    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:46:49.009482 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.009404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:49.009482 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.009453    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:49.009635 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.009511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:49.009635 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.009572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnqb5\" (UniqueName: \"kubernetes.io/projected/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-api-access-jnqb5\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:49.009712 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.009647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:49.010009 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.009975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:49.010215 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.010194    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:49.010301 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.010234    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:49.012059 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.012042    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:49.018069 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.018049    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnqb5\" (UniqueName: \"kubernetes.io/projected/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-api-access-jnqb5\") pod \"isvc-logger-raw-cipn-predictor-787489bd95-l9jfk\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:49.088730 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.088683    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:49.364883 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.364859    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"]
Sep 19 08:46:49.367391 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:46:49.367363    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b74dfdc_a26e_46d4_95fd_3ecf526f5aa8.slice/crio-2c45726b8fcee3472b72162277134613100923627aaa7d432f42b7fc7554d2b9 WatchSource:0}: Error finding container 2c45726b8fcee3472b72162277134613100923627aaa7d432f42b7fc7554d2b9: Status 404 returned error can't find the container with id 2c45726b8fcee3472b72162277134613100923627aaa7d432f42b7fc7554d2b9
Sep 19 08:46:49.739531 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.739437    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" event={"ID":"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8","Type":"ContainerStarted","Data":"f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4"}
Sep 19 08:46:49.739531 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:49.739524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" event={"ID":"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8","Type":"ContainerStarted","Data":"2c45726b8fcee3472b72162277134613100923627aaa7d432f42b7fc7554d2b9"}
Sep 19 08:46:53.757767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:53.757733    2559 generic.go:356] "Generic (PLEG): container finished" podID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerID="f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4" exitCode=0
Sep 19 08:46:53.758172 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:53.757796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" event={"ID":"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8","Type":"ContainerDied","Data":"f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4"}
Sep 19 08:46:54.765075 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:54.765031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" event={"ID":"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8","Type":"ContainerStarted","Data":"a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2"}
Sep 19 08:46:54.765502 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:54.765140    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" event={"ID":"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8","Type":"ContainerStarted","Data":"2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0"}
Sep 19 08:46:54.765502 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:54.765168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" event={"ID":"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8","Type":"ContainerStarted","Data":"c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3"}
Sep 19 08:46:54.765502 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:54.765395    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:54.786622 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:54.786576    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podStartSLOduration=3.021336013 podStartE2EDuration="6.786561998s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.765225985s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:46:54.78536198 +0000 UTC m=+1567.195884982" watchObservedRunningTime="2026-09-19 08:46:54.786561998 +0000 UTC m=+1567.197084997"
Sep 19 08:46:55.769939 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:55.769908    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:55.770507 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:55.769983    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:46:55.771427 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:55.771395    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:46:55.772244 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:55.772216    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:46:56.775487 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:56.775437    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:46:56.775960 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:46:56.775851    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:47:01.781401 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:01.781372    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:47:01.782160 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:01.782126    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:47:01.782328 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:01.782304    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:47:11.782487 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:11.782439    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:47:11.782955 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:11.782933    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:47:21.782366 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:21.782308    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:47:21.782826 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:21.782716    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:47:31.782935 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:31.782838    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:47:31.783463 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:31.783298    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:47:41.782491 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:41.782441    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:47:41.782973 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:41.782902    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:47:51.782524 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:51.782479    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:47:51.782983 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:47:51.782961    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:48:01.782568 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:01.782531    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:48:01.783071 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:01.783044    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:48:11.782804 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:11.782757    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:48:11.783317 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:11.783292    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:48:21.782994 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:21.782961    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:48:21.783442 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:21.783182    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:48:34.216977 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.216937    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-brn8g_3aa543ce-d637-4353-a4b3-c86bec90a8e3/kserve-container/0.log"
Sep 19 08:48:34.454961 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.454922    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"]
Sep 19 08:48:34.455218 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.455181    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" containerID="cri-o://c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3" gracePeriod=30
Sep 19 08:48:34.455401 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.455211    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" containerID="cri-o://a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2" gracePeriod=30
Sep 19 08:48:34.455401 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.455240    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kube-rbac-proxy" containerID="cri-o://2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0" gracePeriod=30
Sep 19 08:48:34.497054 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.497022    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"]
Sep 19 08:48:34.497270 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.497228    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g" podUID="3aa543ce-d637-4353-a4b3-c86bec90a8e3" containerName="kserve-container" containerID="cri-o://14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3" gracePeriod=30
Sep 19 08:48:34.497398 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.497288    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g" podUID="3aa543ce-d637-4353-a4b3-c86bec90a8e3" containerName="kube-rbac-proxy" containerID="cri-o://db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c" gracePeriod=30
Sep 19 08:48:34.500610 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.500574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:48:34.749812 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.749784    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:48:34.863057 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.863025    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r2r7\" (UniqueName: \"kubernetes.io/projected/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-api-access-6r2r7\") pod \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") "
Sep 19 08:48:34.863057 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.863061    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa543ce-d637-4353-a4b3-c86bec90a8e3-proxy-tls\") pod \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") "
Sep 19 08:48:34.863303 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.863095    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-rbac-proxy-sar-config\") pod \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\" (UID: \"3aa543ce-d637-4353-a4b3-c86bec90a8e3\") "
Sep 19 08:48:34.863507 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.863482    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3aa543ce-d637-4353-a4b3-c86bec90a8e3" (UID: "3aa543ce-d637-4353-a4b3-c86bec90a8e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:48:34.865190 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.865170    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aa543ce-d637-4353-a4b3-c86bec90a8e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3aa543ce-d637-4353-a4b3-c86bec90a8e3" (UID: "3aa543ce-d637-4353-a4b3-c86bec90a8e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:48:34.865268 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.865185    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-api-access-6r2r7" (OuterVolumeSpecName: "kube-api-access-6r2r7") pod "3aa543ce-d637-4353-a4b3-c86bec90a8e3" (UID: "3aa543ce-d637-4353-a4b3-c86bec90a8e3"). InnerVolumeSpecName "kube-api-access-6r2r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:48:34.964688 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.964644    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r2r7\" (UniqueName: \"kubernetes.io/projected/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-api-access-6r2r7\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:48:34.964688 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.964680    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3aa543ce-d637-4353-a4b3-c86bec90a8e3-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:48:34.964688 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:34.964695    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3aa543ce-d637-4353-a4b3-c86bec90a8e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:48:35.245330 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.245292    2559 generic.go:356] "Generic (PLEG): container finished" podID="3aa543ce-d637-4353-a4b3-c86bec90a8e3" containerID="db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c" exitCode=2
Sep 19 08:48:35.245330 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.245319    2559 generic.go:356] "Generic (PLEG): container finished" podID="3aa543ce-d637-4353-a4b3-c86bec90a8e3" containerID="14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3" exitCode=2
Sep 19 08:48:35.245903 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.245347    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"
Sep 19 08:48:35.245903 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.245382    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g" event={"ID":"3aa543ce-d637-4353-a4b3-c86bec90a8e3","Type":"ContainerDied","Data":"db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c"}
Sep 19 08:48:35.245903 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.245450    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g" event={"ID":"3aa543ce-d637-4353-a4b3-c86bec90a8e3","Type":"ContainerDied","Data":"14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3"}
Sep 19 08:48:35.245903 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.245480    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g" event={"ID":"3aa543ce-d637-4353-a4b3-c86bec90a8e3","Type":"ContainerDied","Data":"3fc1c790d1790e6f09e8ceb0a64ef32f46699c76d5f42b4ff96b382a59387013"}
Sep 19 08:48:35.245903 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.245491    2559 scope.go:117] "RemoveContainer" containerID="db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c"
Sep 19 08:48:35.248322 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.248282    2559 generic.go:356] "Generic (PLEG): container finished" podID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerID="2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0" exitCode=2
Sep 19 08:48:35.248457 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.248334    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" event={"ID":"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8","Type":"ContainerDied","Data":"2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0"}
Sep 19 08:48:35.254938 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.254923    2559 scope.go:117] "RemoveContainer" containerID="14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3"
Sep 19 08:48:35.264536 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.264496    2559 scope.go:117] "RemoveContainer" containerID="db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c"
Sep 19 08:48:35.264929 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:48:35.264910    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c\": container with ID starting with db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c not found: ID does not exist" containerID="db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c"
Sep 19 08:48:35.264987 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.264938    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c"} err="failed to get container status \"db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c\": rpc error: code = NotFound desc = could not find container \"db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c\": container with ID starting with db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c not found: ID does not exist"
Sep 19 08:48:35.264987 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.264957    2559 scope.go:117] "RemoveContainer" containerID="14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3"
Sep 19 08:48:35.265221 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:48:35.265204    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3\": container with ID starting with 14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3 not found: ID does not exist" containerID="14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3"
Sep 19 08:48:35.265289 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.265227    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3"} err="failed to get container status \"14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3\": rpc error: code = NotFound desc = could not find container \"14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3\": container with ID starting with 14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3 not found: ID does not exist"
Sep 19 08:48:35.265289 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.265244    2559 scope.go:117] "RemoveContainer" containerID="db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c"
Sep 19 08:48:35.265546 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.265527    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c"} err="failed to get container status \"db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c\": rpc error: code = NotFound desc = could not find container \"db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c\": container with ID starting with db3dec10ee1c2dcf034a2d075caa2db4f3a59b20a848e4f07b66d44b5510e41c not found: ID does not exist"
Sep 19 08:48:35.265611 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.265548    2559 scope.go:117] "RemoveContainer" containerID="14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3"
Sep 19 08:48:35.265806 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.265790    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3"} err="failed to get container status \"14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3\": rpc error: code = NotFound desc = could not find container \"14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3\": container with ID starting with 14bd65b42c214b024267f3337a98d3539e6ba4f27b05fdbe6beb88d9625b2ab3 not found: ID does not exist"
Sep 19 08:48:35.269128 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.269106    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"]
Sep 19 08:48:35.272644 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.272623    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-brn8g"]
Sep 19 08:48:35.937994 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:35.937958    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aa543ce-d637-4353-a4b3-c86bec90a8e3" path="/var/lib/kubelet/pods/3aa543ce-d637-4353-a4b3-c86bec90a8e3/volumes"
Sep 19 08:48:36.180059 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:36.180027    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:48:36.776236 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:36.776193    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 19 08:48:40.275810 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:40.275770    2559 generic.go:356] "Generic (PLEG): container finished" podID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerID="c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3" exitCode=0
Sep 19 08:48:40.276206 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:40.275837    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" event={"ID":"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8","Type":"ContainerDied","Data":"c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3"}
Sep 19 08:48:41.775724 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:41.775678    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 19 08:48:41.782331 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:41.782291    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:48:41.782676 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:41.782646    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:48:46.776537 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:46.776486    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 19 08:48:46.777036 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:46.776645    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:48:50.861087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:50.861055    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:50.883396 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:50.883366    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:50.892062 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:50.892037    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:50.901579 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:50.901560    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:48:51.684890 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:51.684858    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:51.696469 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:51.696437    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:51.707088 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:51.707067    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:51.717728 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:51.717702    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:48:51.775846 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:51.775805    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 19 08:48:51.782300 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:51.782249    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:48:51.782678 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:51.782652    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:48:52.522712 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:52.522679    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:52.533652 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:52.533623    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:52.542717 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:52.542691    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:52.552722 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:52.552697    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:48:53.323709 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:53.323677    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:53.334416 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:53.334379    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:53.342998 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:53.342963    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:53.351622 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:53.351601    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:48:54.154369 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:54.154330    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:54.165512 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:54.165480    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:54.174990 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:54.174966    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:54.184416 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:54.184395    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:48:54.956579 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:54.956544    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:54.967767 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:54.967741    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:54.976710 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:54.976688    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:54.986167 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:54.986136    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:48:55.748828 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:55.748798    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:55.759650 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:55.759620    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:55.767927 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:55.767904    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:55.776921 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:55.776893    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:48:56.534161 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:56.534107    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:56.544952 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:56.544929    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:56.553780 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:56.553754    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:56.562634 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:56.562598    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:48:56.775573 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:56.775531    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 19 08:48:57.350996 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:57.350966    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:57.361591 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:57.361548    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:57.369983 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:57.369957    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:57.379786 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:57.379764    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:48:58.161358 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:58.161324    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:58.172620 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:58.172587    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:58.181502 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:58.181477    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:58.190967 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:58.190946    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:48:58.979241 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:58.979208    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:58.990140 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:58.990108    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:58.998828 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:58.998807    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:59.007991 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:59.007967    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:48:59.781947 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:59.781913    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:48:59.792443 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:59.792420    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:48:59.801305 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:59.801271    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:48:59.810218 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:48:59.810196    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:49:00.579987 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:00.579958    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:49:00.590085 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:00.590052    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:49:00.599600 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:00.599573    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:49:00.608161 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:00.608138    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:49:01.377963 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:01.377933    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:49:01.389025 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:01.389004    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:49:01.403487 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:01.403458    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:49:01.413115 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:01.413093    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:49:01.776632 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:01.776542    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 19 08:49:01.782035 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:01.781996    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 19 08:49:01.782199 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:01.782125    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:49:01.782620 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:01.782594    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 19 08:49:01.782710 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:01.782702    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:49:02.273307 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:02.273276    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kserve-container/0.log"
Sep 19 08:49:02.285780 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:02.285752    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/kube-rbac-proxy/0.log"
Sep 19 08:49:02.296360 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:02.296334    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/agent/0.log"
Sep 19 08:49:02.306012 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:02.305985    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-787489bd95-l9jfk_3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/storage-initializer/0.log"
Sep 19 08:49:04.603237 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.603209    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:49:04.722161 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.722066    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnqb5\" (UniqueName: \"kubernetes.io/projected/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-api-access-jnqb5\") pod \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") "
Sep 19 08:49:04.722161 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.722117    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kserve-provision-location\") pod \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") "
Sep 19 08:49:04.722161 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.722140    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-proxy-tls\") pod \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") "
Sep 19 08:49:04.722444 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.722168    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-cabundle-cert\") pod \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") "
Sep 19 08:49:04.722444 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.722200    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-rbac-proxy-sar-config\") pod \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\" (UID: \"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8\") "
Sep 19 08:49:04.722600 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.722499    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" (UID: "3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:04.722600 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.722594    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" (UID: "3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:49:04.722757 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.722630    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" (UID: "3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:49:04.724325 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.724308    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-api-access-jnqb5" (OuterVolumeSpecName: "kube-api-access-jnqb5") pod "3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" (UID: "3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8"). InnerVolumeSpecName "kube-api-access-jnqb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:49:04.724463 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.724444    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" (UID: "3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:49:04.823694 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.823658    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnqb5\" (UniqueName: \"kubernetes.io/projected/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-api-access-jnqb5\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:49:04.823694 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.823687    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kserve-provision-location\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:49:04.823694 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.823697    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-proxy-tls\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:49:04.823694 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.823708    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-cabundle-cert\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:49:04.823965 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:04.823716    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-26.ec2.internal\" DevicePath \"\""
Sep 19 08:49:05.392433 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.392392    2559 generic.go:356] "Generic (PLEG): container finished" podID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerID="a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2" exitCode=0
Sep 19 08:49:05.392753 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.392488    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"
Sep 19 08:49:05.392753 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.392481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" event={"ID":"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8","Type":"ContainerDied","Data":"a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2"}
Sep 19 08:49:05.392753 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.392628    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk" event={"ID":"3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8","Type":"ContainerDied","Data":"2c45726b8fcee3472b72162277134613100923627aaa7d432f42b7fc7554d2b9"}
Sep 19 08:49:05.392753 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.392660    2559 scope.go:117] "RemoveContainer" containerID="a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2"
Sep 19 08:49:05.401766 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.401750    2559 scope.go:117] "RemoveContainer" containerID="2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0"
Sep 19 08:49:05.409575 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.409560    2559 scope.go:117] "RemoveContainer" containerID="c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3"
Sep 19 08:49:05.418130 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.418103    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"]
Sep 19 08:49:05.418665 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.418630    2559 scope.go:117] "RemoveContainer" containerID="f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4"
Sep 19 08:49:05.420918 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.420895    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-787489bd95-l9jfk"]
Sep 19 08:49:05.426563 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.426549    2559 scope.go:117] "RemoveContainer" containerID="a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2"
Sep 19 08:49:05.426835 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:49:05.426819    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2\": container with ID starting with a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2 not found: ID does not exist" containerID="a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2"
Sep 19 08:49:05.426884 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.426843    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2"} err="failed to get container status \"a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2\": rpc error: code = NotFound desc = could not find container \"a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2\": container with ID starting with a5533ec72a4a8b6b910abd3ca9eb27ccc11139a726bbf36af1f904aa74be14e2 not found: ID does not exist"
Sep 19 08:49:05.426884 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.426863    2559 scope.go:117] "RemoveContainer" containerID="2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0"
Sep 19 08:49:05.427108 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:49:05.427089    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0\": container with ID starting with 2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0 not found: ID does not exist" containerID="2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0"
Sep 19 08:49:05.427152 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.427117    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0"} err="failed to get container status \"2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0\": rpc error: code = NotFound desc = could not find container \"2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0\": container with ID starting with 2b2f8bf2c502b88966b388d65e24714194df9338a6af5276001c701ef36417a0 not found: ID does not exist"
Sep 19 08:49:05.427152 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.427133    2559 scope.go:117] "RemoveContainer" containerID="c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3"
Sep 19 08:49:05.427383 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:49:05.427366    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3\": container with ID starting with c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3 not found: ID does not exist" containerID="c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3"
Sep 19 08:49:05.427435 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.427389    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3"} err="failed to get container status \"c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3\": rpc error: code = NotFound desc = could not find container \"c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3\": container with ID starting with c204fbe68d39bdbe482ba5a7c027d3e4755c52cd28c3974afb4bbe97cf1761c3 not found: ID does not exist"
Sep 19 08:49:05.427435 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.427402    2559 scope.go:117] "RemoveContainer" containerID="f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4"
Sep 19 08:49:05.427637 ip-10-0-134-26 kubenswrapper[2559]: E0919 08:49:05.427620    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4\": container with ID starting with f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4 not found: ID does not exist" containerID="f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4"
Sep 19 08:49:05.427680 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.427644    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4"} err="failed to get container status \"f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4\": rpc error: code = NotFound desc = could not find container \"f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4\": container with ID starting with f80e13b372312f86ef796f159d94eba4a6c7466a59bd0dfae9a1165b2ea348b4 not found: ID does not exist"
Sep 19 08:49:05.937484 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:05.937448    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" path="/var/lib/kubelet/pods/3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8/volumes"
Sep 19 08:49:06.935976 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:06.935941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:49:07.074361 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074317    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lnnkg/must-gather-vzkf4"]
Sep 19 08:49:07.074735 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074665    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container"
Sep 19 08:49:07.074735 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074677    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container"
Sep 19 08:49:07.074735 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074689    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa543ce-d637-4353-a4b3-c86bec90a8e3" containerName="kserve-container"
Sep 19 08:49:07.074735 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074695    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa543ce-d637-4353-a4b3-c86bec90a8e3" containerName="kserve-container"
Sep 19 08:49:07.074735 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074704    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa543ce-d637-4353-a4b3-c86bec90a8e3" containerName="kube-rbac-proxy"
Sep 19 08:49:07.074735 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074711    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa543ce-d637-4353-a4b3-c86bec90a8e3" containerName="kube-rbac-proxy"
Sep 19 08:49:07.074735 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074722    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent"
Sep 19 08:49:07.074735 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074727    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent"
Sep 19 08:49:07.074735 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074739    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="storage-initializer"
Sep 19 08:49:07.075029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074744    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="storage-initializer"
Sep 19 08:49:07.075029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074751    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kube-rbac-proxy"
Sep 19 08:49:07.075029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074756    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kube-rbac-proxy"
Sep 19 08:49:07.075029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074811    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="agent"
Sep 19 08:49:07.075029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074818    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa543ce-d637-4353-a4b3-c86bec90a8e3" containerName="kserve-container"
Sep 19 08:49:07.075029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074826    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa543ce-d637-4353-a4b3-c86bec90a8e3" containerName="kube-rbac-proxy"
Sep 19 08:49:07.075029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074834    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kube-rbac-proxy"
Sep 19 08:49:07.075029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.074842    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b74dfdc-a26e-46d4-95fd-3ecf526f5aa8" containerName="kserve-container"
Sep 19 08:49:07.079492 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.079471    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lnnkg/must-gather-vzkf4"
Sep 19 08:49:07.082124 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.082099    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lnnkg\"/\"openshift-service-ca.crt\""
Sep 19 08:49:07.082124 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.082119    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lnnkg\"/\"default-dockercfg-gczgz\""
Sep 19 08:49:07.082362 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.082108    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lnnkg\"/\"kube-root-ca.crt\""
Sep 19 08:49:07.087819 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.087795    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lnnkg/must-gather-vzkf4"]
Sep 19 08:49:07.093367 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.093327    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:49:07.144106 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.144067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvnfj\" (UniqueName: \"kubernetes.io/projected/699a5937-d105-4a24-821b-9fde0f76e702-kube-api-access-zvnfj\") pod \"must-gather-vzkf4\" (UID: \"699a5937-d105-4a24-821b-9fde0f76e702\") " pod="openshift-must-gather-lnnkg/must-gather-vzkf4"
Sep 19 08:49:07.144302 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.144176    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/699a5937-d105-4a24-821b-9fde0f76e702-must-gather-output\") pod \"must-gather-vzkf4\" (UID: \"699a5937-d105-4a24-821b-9fde0f76e702\") " pod="openshift-must-gather-lnnkg/must-gather-vzkf4"
Sep 19 08:49:07.194608 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.194516    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:49:07.244956 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.244920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/699a5937-d105-4a24-821b-9fde0f76e702-must-gather-output\") pod \"must-gather-vzkf4\" (UID: \"699a5937-d105-4a24-821b-9fde0f76e702\") " pod="openshift-must-gather-lnnkg/must-gather-vzkf4"
Sep 19 08:49:07.245128 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.244980    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvnfj\" (UniqueName: \"kubernetes.io/projected/699a5937-d105-4a24-821b-9fde0f76e702-kube-api-access-zvnfj\") pod \"must-gather-vzkf4\" (UID: \"699a5937-d105-4a24-821b-9fde0f76e702\") " pod="openshift-must-gather-lnnkg/must-gather-vzkf4"
Sep 19 08:49:07.245249 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.245232    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/699a5937-d105-4a24-821b-9fde0f76e702-must-gather-output\") pod \"must-gather-vzkf4\" (UID: \"699a5937-d105-4a24-821b-9fde0f76e702\") " pod="openshift-must-gather-lnnkg/must-gather-vzkf4"
Sep 19 08:49:07.253058 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.253028    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvnfj\" (UniqueName: \"kubernetes.io/projected/699a5937-d105-4a24-821b-9fde0f76e702-kube-api-access-zvnfj\") pod \"must-gather-vzkf4\" (UID: \"699a5937-d105-4a24-821b-9fde0f76e702\") " pod="openshift-must-gather-lnnkg/must-gather-vzkf4"
Sep 19 08:49:07.397317 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.397280    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lnnkg/must-gather-vzkf4"
Sep 19 08:49:07.667387 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.667359    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lnnkg/must-gather-vzkf4"]
Sep 19 08:49:07.669922 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:49:07.669896    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod699a5937_d105_4a24_821b_9fde0f76e702.slice/crio-e1565d9cb48a8cf60010f78cbe2a807326114ef46df4593b811c1c306cfc12bb WatchSource:0}: Error finding container e1565d9cb48a8cf60010f78cbe2a807326114ef46df4593b811c1c306cfc12bb: Status 404 returned error can't find the container with id e1565d9cb48a8cf60010f78cbe2a807326114ef46df4593b811c1c306cfc12bb
Sep 19 08:49:07.671786 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:07.671769    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:49:08.410131 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:08.410090    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lnnkg/must-gather-vzkf4" event={"ID":"699a5937-d105-4a24-821b-9fde0f76e702","Type":"ContainerStarted","Data":"e1565d9cb48a8cf60010f78cbe2a807326114ef46df4593b811c1c306cfc12bb"}
Sep 19 08:49:09.419879 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:09.419840    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lnnkg/must-gather-vzkf4" event={"ID":"699a5937-d105-4a24-821b-9fde0f76e702","Type":"ContainerStarted","Data":"f9e3ecad680f4f549b8a46248cc48b5a260ab7e21c0b9034d330be34a5592d93"}
Sep 19 08:49:09.420345 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:09.419896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lnnkg/must-gather-vzkf4" event={"ID":"699a5937-d105-4a24-821b-9fde0f76e702","Type":"ContainerStarted","Data":"7d73bfff5936c868e38b3cb8926d94854162bd15578f4f316209de82eb751808"}
Sep 19 08:49:09.438021 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:09.437970    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lnnkg/must-gather-vzkf4" podStartSLOduration=1.644232119 podStartE2EDuration="2.437954803s" totalImagesPullingTime="793.722684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:49:09.435439838 +0000 UTC m=+1701.845962839" watchObservedRunningTime="2026-09-19 08:49:09.437954803 +0000 UTC m=+1701.848477834"
Sep 19 08:49:09.945272 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:09.945233    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xvkt5_078bb266-8297-48c9-a45b-2d47fb7cbd45/global-pull-secret-syncer/0.log"
Sep 19 08:49:09.984794 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:09.984756    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log"
Sep 19 08:49:10.074559 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:10.074524    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-26.ec2.internal_3a7baa0c5fc4f5272bafad3e803d0c95/haproxy/0.log"
Sep 19 08:49:12.449172 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:12.449135    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:49:12.515556 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:12.515502    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-h46xh_955095d1-46e8-49b9-a667-a40d58fa50ef/konnectivity-agent/0.log" limit=1024
Sep 19 08:49:14.353049 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.353024    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8aa0250-06dd-4d6f-b4dc-880851872780/alertmanager/0.log"
Sep 19 08:49:14.381087 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.381050    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8aa0250-06dd-4d6f-b4dc-880851872780/config-reloader/0.log"
Sep 19 08:49:14.406590 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.406555    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8aa0250-06dd-4d6f-b4dc-880851872780/kube-rbac-proxy-web/0.log"
Sep 19 08:49:14.428701 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.428667    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8aa0250-06dd-4d6f-b4dc-880851872780/kube-rbac-proxy/0.log"
Sep 19 08:49:14.454851 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.454814    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8aa0250-06dd-4d6f-b4dc-880851872780/kube-rbac-proxy-metric/0.log"
Sep 19 08:49:14.487731 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.487705    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8aa0250-06dd-4d6f-b4dc-880851872780/prom-label-proxy/0.log"
Sep 19 08:49:14.525215 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.525184    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d8aa0250-06dd-4d6f-b4dc-880851872780/init-config-reloader/0.log"
Sep 19 08:49:14.590992 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.590965    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-dpq85_11edfda1-bde6-4329-b746-655d3a1cd773/kube-state-metrics/0.log"
Sep 19 08:49:14.619821 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.619737    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-dpq85_11edfda1-bde6-4329-b746-655d3a1cd773/kube-rbac-proxy-main/0.log"
Sep 19 08:49:14.640106 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.640061    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-dpq85_11edfda1-bde6-4329-b746-655d3a1cd773/kube-rbac-proxy-self/0.log"
Sep 19 08:49:14.831982 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.831949    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fplhj_61795267-8b75-45e6-8236-8478035beaed/node-exporter/0.log"
Sep 19 08:49:14.865283 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.863336    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fplhj_61795267-8b75-45e6-8236-8478035beaed/kube-rbac-proxy/0.log"
Sep 19 08:49:14.890956 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:14.890860    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fplhj_61795267-8b75-45e6-8236-8478035beaed/init-textfile/0.log"
Sep 19 08:49:15.013692 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:15.013660    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-wxndl_011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f/kube-rbac-proxy-main/0.log"
Sep 19 08:49:15.039710 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:15.039669    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-wxndl_011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f/kube-rbac-proxy-self/0.log"
Sep 19 08:49:15.060636 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:15.060568    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-wxndl_011ee6eb-d2a8-4c57-b205-2b3f31c2ad7f/openshift-state-metrics/0.log"
Sep 19 08:49:15.333032 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:15.332999    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56d65b9b7f-q9xtm_ff541a40-d461-4c09-8d9e-a99bfa37a0a6/telemeter-client/0.log"
Sep 19 08:49:15.354583 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:15.354559    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56d65b9b7f-q9xtm_ff541a40-d461-4c09-8d9e-a99bfa37a0a6/reload/0.log"
Sep 19 08:49:15.376785 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:15.376755    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56d65b9b7f-q9xtm_ff541a40-d461-4c09-8d9e-a99bfa37a0a6/kube-rbac-proxy/0.log"
Sep 19 08:49:17.780328 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:17.780277    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-64b9cc5599-s69h8_d2794acc-d339-4f6c-a2d3-ff37cb3ca9a9/console/0.log"
Sep 19 08:49:17.813660 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:17.813619    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-bqfng_d335ad1a-3686-4a3f-9ef8-88f7989199e6/download-server/0.log"
Sep 19 08:49:18.318727 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.318697    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-gfznx_cfe03657-f8db-4e34-90f3-e6762a4f8753/volume-data-source-validator/0.log"
Sep 19 08:49:18.327020 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.326990    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"]
Sep 19 08:49:18.332591 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.332566    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.341411 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.341384    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"]
Sep 19 08:49:18.459759 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.459730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-lib-modules\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.459962 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.459781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-podres\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.459962 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.459905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2brg7\" (UniqueName: \"kubernetes.io/projected/5d5ebd31-739e-4845-bbd5-66d80255b813-kube-api-access-2brg7\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.460095 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.459961    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-proc\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.460095 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.460001    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-sys\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.561588 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.561548    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-proc\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.561962 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.561944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-sys\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.562187 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.562173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-lib-modules\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.562498 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.562482    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-podres\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.562766 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.562751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2brg7\" (UniqueName: \"kubernetes.io/projected/5d5ebd31-739e-4845-bbd5-66d80255b813-kube-api-access-2brg7\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.563221 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.562416    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-lib-modules\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.563383 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.562099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-sys\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.563476 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.562668    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-podres\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.563559 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.561873    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d5ebd31-739e-4845-bbd5-66d80255b813-proc\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.570934 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.570872    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2brg7\" (UniqueName: \"kubernetes.io/projected/5d5ebd31-739e-4845-bbd5-66d80255b813-kube-api-access-2brg7\") pod \"perf-node-gather-daemonset-vglqf\" (UID: \"5d5ebd31-739e-4845-bbd5-66d80255b813\") " pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.646696 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.646661    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:18.941419 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:18.940808    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"]
Sep 19 08:49:18.945962 ip-10-0-134-26 kubenswrapper[2559]: W0919 08:49:18.945915    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5d5ebd31_739e_4845_bbd5_66d80255b813.slice/crio-4222c0b9f8c060b4dfe65c6ea0dfe3e0a9043f3fabbaa4e002db570c35ccadda WatchSource:0}: Error finding container 4222c0b9f8c060b4dfe65c6ea0dfe3e0a9043f3fabbaa4e002db570c35ccadda: Status 404 returned error can't find the container with id 4222c0b9f8c060b4dfe65c6ea0dfe3e0a9043f3fabbaa4e002db570c35ccadda
Sep 19 08:49:19.315272 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:19.315212    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-svmsl_6663b01e-cdec-4748-a1b3-7fd84903131d/dns/0.log"
Sep 19 08:49:19.335635 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:19.335607    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-svmsl_6663b01e-cdec-4748-a1b3-7fd84903131d/kube-rbac-proxy/0.log"
Sep 19 08:49:19.406646 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:19.406612    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xn4mf_af211ba6-f358-4565-a7dd-37eb2337ea07/dns-node-resolver/0.log"
Sep 19 08:49:19.489513 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:19.489472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf" event={"ID":"5d5ebd31-739e-4845-bbd5-66d80255b813","Type":"ContainerStarted","Data":"6d6c7d59ec5f811ba70a30dcd9a4061622095950a30495c16b5352dac7d0772a"}
Sep 19 08:49:19.489725 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:19.489541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf" event={"ID":"5d5ebd31-739e-4845-bbd5-66d80255b813","Type":"ContainerStarted","Data":"4222c0b9f8c060b4dfe65c6ea0dfe3e0a9043f3fabbaa4e002db570c35ccadda"}
Sep 19 08:49:19.490348 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:19.490315    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:19.507187 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:19.507131    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf" podStartSLOduration=1.507111479 podStartE2EDuration="1.507111479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:49:19.505962748 +0000 UTC m=+1711.916485750" watchObservedRunningTime="2026-09-19 08:49:19.507111479 +0000 UTC m=+1711.917634481"
Sep 19 08:49:19.866116 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:19.866077    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c6758476b-lz964_a03882a4-c7f1-41d9-932b-54efda96e8ab/registry/0.log"
Sep 19 08:49:19.907465 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:19.907417    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lcmrp_2d618e83-8b27-4401-94d2-1feaa04c634f/node-ca/0.log"
Sep 19 08:49:20.941527 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:20.941496    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-45sdz_8098f93f-f309-4e66-a1a5-fe28c2b8a5d8/serve-healthcheck-canary/0.log"
Sep 19 08:49:21.360659 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:21.360620    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-hhscx_60dc08bc-8f96-458d-a5a7-c09534ca4ef5/insights-operator/0.log"
Sep 19 08:49:21.361203 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:21.361173    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-hhscx_60dc08bc-8f96-458d-a5a7-c09534ca4ef5/insights-operator/1.log"
Sep 19 08:49:21.438906 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:21.438875    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t7vcg_c64a1e78-7b83-4797-baca-6f3ccbea5a53/kube-rbac-proxy/0.log"
Sep 19 08:49:21.457246 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:21.457216    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t7vcg_c64a1e78-7b83-4797-baca-6f3ccbea5a53/exporter/0.log"
Sep 19 08:49:21.475956 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:21.475928    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t7vcg_c64a1e78-7b83-4797-baca-6f3ccbea5a53/extractor/0.log"
Sep 19 08:49:23.802033 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:23.802007    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-dqlqz_008de914-d181-4d8c-83a2-dd742baa733b/server/0.log"
Sep 19 08:49:24.029863 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:24.029825    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-b5vsn_5b1a44c2-0af7-4b1a-837c-00a9327ff5b3/manager/0.log"
Sep 19 08:49:24.073841 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:24.073761    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vtmbg_ded8cbde-4d6c-4edb-a5bd-6df62838e3f7/s3-init/0.log"
Sep 19 08:49:24.088585 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:24.088559    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vtmbg_ded8cbde-4d6c-4edb-a5bd-6df62838e3f7/download-hf-model/0.log"
Sep 19 08:49:24.122350 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:24.122313    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-w2vfx_d36d14a0-5430-4555-ab10-4b151f2d7bc8/seaweedfs/0.log"
Sep 19 08:49:26.509038 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:26.509010    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lnnkg/perf-node-gather-daemonset-vglqf"
Sep 19 08:49:28.142337 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:28.142297    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-k7g64_af453c92-08c7-44c0-9e10-52cf98b6b7f4/kube-storage-version-migrator-operator/1.log"
Sep 19 08:49:28.142864 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:28.142746    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-k7g64_af453c92-08c7-44c0-9e10-52cf98b6b7f4/kube-storage-version-migrator-operator/0.log"
Sep 19 08:49:29.322803 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:29.322717    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2x69_9fb5bd3b-10c8-41fb-b8b1-d14846935295/kube-multus-additional-cni-plugins/0.log"
Sep 19 08:49:29.344209 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:29.344171    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2x69_9fb5bd3b-10c8-41fb-b8b1-d14846935295/egress-router-binary-copy/0.log"
Sep 19 08:49:29.368175 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:29.368142    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2x69_9fb5bd3b-10c8-41fb-b8b1-d14846935295/cni-plugins/0.log"
Sep 19 08:49:29.392716 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:29.392678    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2x69_9fb5bd3b-10c8-41fb-b8b1-d14846935295/bond-cni-plugin/0.log"
Sep 19 08:49:29.412408 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:29.412374    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2x69_9fb5bd3b-10c8-41fb-b8b1-d14846935295/routeoverride-cni/0.log"
Sep 19 08:49:29.433603 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:29.433577    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2x69_9fb5bd3b-10c8-41fb-b8b1-d14846935295/whereabouts-cni-bincopy/0.log"
Sep 19 08:49:29.454149 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:29.454120    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2x69_9fb5bd3b-10c8-41fb-b8b1-d14846935295/whereabouts-cni/0.log"
Sep 19 08:49:29.693234 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:29.693162    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pv57b_007b6a79-68a9-4bf6-8d49-897b5151c149/kube-multus/0.log"
Sep 19 08:49:29.715279 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:29.715219    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7qn5v_9178e60e-1f54-461c-8a40-1b8c3300b64c/network-metrics-daemon/0.log"
Sep 19 08:49:29.734076 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:29.734053    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7qn5v_9178e60e-1f54-461c-8a40-1b8c3300b64c/kube-rbac-proxy/0.log"
Sep 19 08:49:30.974484 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:30.974448    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnft5_338a6f68-5bed-457b-b1f2-23f4e68922d6/ovn-controller/0.log"
Sep 19 08:49:30.990755 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:30.990727    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnft5_338a6f68-5bed-457b-b1f2-23f4e68922d6/ovn-acl-logging/0.log"
Sep 19 08:49:31.028029 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:31.027992    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnft5_338a6f68-5bed-457b-b1f2-23f4e68922d6/kube-rbac-proxy-node/0.log"
Sep 19 08:49:31.051129 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:31.051049    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnft5_338a6f68-5bed-457b-b1f2-23f4e68922d6/kube-rbac-proxy-ovn-metrics/0.log"
Sep 19 08:49:31.067343 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:31.067317    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnft5_338a6f68-5bed-457b-b1f2-23f4e68922d6/northd/0.log"
Sep 19 08:49:31.089413 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:31.089383    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnft5_338a6f68-5bed-457b-b1f2-23f4e68922d6/nbdb/0.log"
Sep 19 08:49:31.108842 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:31.108815    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnft5_338a6f68-5bed-457b-b1f2-23f4e68922d6/sbdb/0.log"
Sep 19 08:49:31.170317 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:31.170280    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnft5_338a6f68-5bed-457b-b1f2-23f4e68922d6/ovnkube-controller/0.log"
Sep 19 08:49:32.104599 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:32.104563    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-gcm28_4ca7dd96-1b7a-462b-9d32-8c9f89fea1e1/network-check-target-container/0.log"
Sep 19 08:49:33.004838 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:33.004812    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-wnn76_8ac0106e-5a03-4fcb-a733-5fd551497dc0/iptables-alerter/0.log"
Sep 19 08:49:33.616977 ip-10-0-134-26 kubenswrapper[2559]: I0919 08:49:33.616942    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-tnb9c_bca89dd8-76bb-4909-be5e-91f7f7fa40bc/tuned/0.log"