Sep 09 20:09:12.732169 ip-10-0-132-109 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:09:12.732180 ip-10-0-132-109 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:09:12.732187 ip-10-0-132-109 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:09:12.732426 ip-10-0-132-109 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:09:22.159635 ip-10-0-132-109 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 09 20:09:22.159646 ip-10-0-132-109 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 09fcaef53d2948aeb47453a3ced90ecb --
Sep 09 20:11:05.495430 ip-10-0-132-109 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:11:05.937066 ip-10-0-132-109 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:05.937066 ip-10-0-132-109 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:11:05.937066 ip-10-0-132-109 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:05.937640 ip-10-0-132-109 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:11:05.937640 ip-10-0-132-109 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:11:05.939000 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.938922    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:11:05.942715 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942699    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942716    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942721    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942728    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942730    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942733    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942736    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942738    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942741    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942746    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942750    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942753    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942756    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:05.942754 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942759    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942763    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942765    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942768    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942771    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942774    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942777    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942779    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942782    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942784    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942786    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942789    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942791    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942794    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942796    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942798    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942801    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942803    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942806    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:05.943053 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942808    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942811    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942813    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942816    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942819    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942821    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942823    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942826    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942828    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942831    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942834    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942836    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942839    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942843    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942846    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942850    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942853    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942855    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942858    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:05.943548 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942860    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942863    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942866    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942868    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942871    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942873    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942876    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942878    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942881    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942883    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942886    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942889    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942891    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942897    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942900    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942903    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942905    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942908    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942910    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942913    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:05.944005 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942915    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942918    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942920    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942923    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942925    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942928    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942930    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942933    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942936    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942938    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942941    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942945    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942948    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942951    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942954    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942956    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942959    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942961    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942964    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942966    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:05.944583 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942969    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:05.945057 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942971    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:05.945057 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942973    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:05.945057 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942976    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:05.945057 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942978    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:05.945057 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942980    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:05.945057 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942983    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:05.945057 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.942985    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:05.947273 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947257    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:05.947273 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947270    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:05.947273 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947273    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:05.947273 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947277    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947279    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947282    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947285    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947288    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947291    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947294    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947297    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947300    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947302    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947306    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947308    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947311    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947313    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947316    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947318    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947321    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947323    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947326    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947328    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:05.947415 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947330    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947333    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947336    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947338    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947340    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947343    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947345    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947348    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947350    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947353    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947355    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947358    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947360    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947363    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947366    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947371    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947375    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947378    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947381    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947383    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:05.950773 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947386    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947389    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947392    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947395    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947397    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947400    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947403    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947405    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947408    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947412    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947415    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947418    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947421    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947425    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947429    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947433    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947436    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947439    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947442    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:05.956890 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947444    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947447    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947449    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947452    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947455    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947458    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947461    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947464    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947467    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947470    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947472    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947475    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947478    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947480    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947483    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947485    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947488    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947490    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947493    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:05.957386 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947495    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947499    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947502    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947505    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947507    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947510    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947512    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947514    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947517    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947519    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947522    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947524    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947527    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947529    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947532    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947534    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947537    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947539    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947923    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947928    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:05.957851 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947932    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947936    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947939    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947941    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947944    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947946    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947949    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947952    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947955    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947957    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947960    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947962    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947964    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947967    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947970    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947973    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947975    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947978    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947980    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:05.958381 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947983    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947985    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947987    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947990    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947992    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947995    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947997    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.947999    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948002    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948004    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948007    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948009    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948012    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948014    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948017    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948020    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948022    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948025    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948027    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948030    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:05.958835 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948032    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948037    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948040    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948043    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948045    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948048    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948050    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948052    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948055    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948058    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948060    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948062    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948065    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948067    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948069    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948072    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948074    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948076    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948079    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948081    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:05.959319 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948084    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948086    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948088    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948092    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948095    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948099    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948102    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948104    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948107    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948110    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948112    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948115    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948117    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948120    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948122    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948139    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948142    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948145    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948149    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:05.959788 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948153    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948156    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948159    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948162    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948164    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948167    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948169    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948172    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948174    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948177    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948179    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948182    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948184    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948187    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948190    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948192    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948195    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948197    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948200    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948271    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:11:05.960290 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948306    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948312    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948316    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948320    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948323    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948327    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948331    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948334    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948337    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948340    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948343    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948346    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948349    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948360    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948363    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948367    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948370    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948374    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948377    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948380    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948383    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948386    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948392    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:11:05.960792 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948395    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948398    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948401    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948404    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948407    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948410    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948432    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948435    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948439    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948442    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948445    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948448    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948450    2579 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948453    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948457    2579 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948460    2579 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948463    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948465    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948468    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948471    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948474    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948477    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948480    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948483    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948486    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:11:05.961347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948489    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948492    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948495    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948497    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948500    2579 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948504    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948510    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948513    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948516    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948519    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948522    2579 flags.go:64] FLAG: --help="false"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948525    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-132-109.ec2.internal"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948529    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948532    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948535    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948538    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948541    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948544    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948547    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948550    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948552    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948555    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948558    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948561    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:11:05.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948564    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948566    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948569    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948572    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948574    2579 flags.go:64] FLAG: --lock-file=""
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948577    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948580    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948583    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948588    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948591    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948594    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948597    2579 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948599    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948603    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948606    2579 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948609    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948613    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948616    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948619    2579 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948622    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948626    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948629    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948632    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948635    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948638    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:11:05.962537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948641    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948648    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948650    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948653    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948656    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948659    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948664    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948667    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948670    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948673    2579 flags.go:64] FLAG: --port="10250"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948676    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948679    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09fe61b67713d2caf"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948682    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948685    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948688    2579 flags.go:64] FLAG: --register-node="true"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948691    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948694    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948697    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948700    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948703    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948706    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948711    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948714    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948716    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948719    2579 flags.go:64] FLAG: --runonce="false"
Sep 09 20:11:05.963118 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948722    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948725    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948728    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948732    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948735    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948738    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948741    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948744    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948746    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948749    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948752    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948755    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948758    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948760    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948763    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948768    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948771    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948774    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948778    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948780    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948783    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948786    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948789    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948791    2579 flags.go:64] FLAG: --v="2"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948795    2579 flags.go:64] FLAG: --version="false"
Sep 09 20:11:05.963762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948799    2579 flags.go:64] FLAG: --vmodule=""
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948802    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.948805    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948911    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948914    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948917    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948921    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948924    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948927    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948929    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948933    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948936    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948939    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948942    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948944    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948947    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948950    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948952    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948955    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948957    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:05.964380 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948960    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948963    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948966    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948968    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948971    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948973    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948976    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948978    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948981    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948983    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948986    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948988    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948991    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948994    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.948996    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949000    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949002    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949005    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949008    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949010    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:05.964881 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949013    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949015    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949019    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949021    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949024    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949027    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949030    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949032    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949035    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949039    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949042    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949044    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949047    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949049    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949052    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949055    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949057    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949060    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949062    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949065    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:05.965458 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949067    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949070    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949072    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949075    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949077    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949080    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949083    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949087    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949091    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949095    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949098    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949101    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949103    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949106    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949110    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949113    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949116    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949118    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949121    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:05.965939 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949140    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949143    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949146    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949148    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949151    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949154    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949156    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949159    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949161    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949164    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949166    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949169    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949172    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949174    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949176    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949179    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949182    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949184    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949187    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:05.966403 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949189    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949194    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949198    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.949201    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.949949    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.958362    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.958376    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.959304    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.959312    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959377    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959383    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959386    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959389    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959392    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:05.966886 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959395    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959398    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959401    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959403    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959406    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959408    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959410    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959413    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959415    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959418    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959421    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959424    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959427    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959429    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959431    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959434    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959437    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959439    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959442    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959444    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:05.967256 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959447    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959450    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959453    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959455    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959459    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959462    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959464    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959468    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959472    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959475    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959478    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959480    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959483    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959486    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959489    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959492    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959494    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959497    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959500    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:05.967733 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959502    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959505    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959507    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959510    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959512    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959515    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959518    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959520    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959523    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959526    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959528    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959531    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959533    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959536    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959539    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959543    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959546    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959549    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959552    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959555    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:05.968188 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959557    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959560    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959562    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959565    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959567    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959570    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959572    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959575    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959577    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959580    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959582    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959585    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959587    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959589    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959592    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959594    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959597    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959600    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959602    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959604    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:05.968694 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959607    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959610    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959613    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959617    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959619    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959622    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959624    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959627    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959629    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959632    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959634    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959637    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959639    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959642    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959644    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:05.969205 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.959649    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959795    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959801    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959804    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959806    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959809    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959812    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959814    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959816    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959819    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959821    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959824    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959827    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959829    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959831    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959834    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959837    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959839    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959842    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959844    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:11:05.969592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959846    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959850    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959854    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959857    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959860    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959863    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959866    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959870    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959873    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959877    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959880    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959882    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959885    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959887    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959889    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959892    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959894    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959896    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959899    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959901    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:11:05.970051 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959903    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959906    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959908    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959910    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959913    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959915    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959918    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959920    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959923    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959926    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959928    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959930    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959933    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959935    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959937    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959940    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959942    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959945    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959948    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959950    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:11:05.970535 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959952    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959955    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959958    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959960    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959962    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959965    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959967    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959969    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959972    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959974    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959976    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959979    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959981    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959983    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959986    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959988    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959990    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959993    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959995    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:11:05.971007 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.959998    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960001    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960003    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960005    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960008    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960010    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960013    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960015    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960018    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960020    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960023    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960025    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960028    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960030    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960033    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960036    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960045    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960048    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960051    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:11:05.971461 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960054    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:11:05.971929 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:05.960056    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:11:05.971929 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.960061    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:11:05.971929 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.960186    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:11:05.971929 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.962457    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:11:05.971929 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.963512    2579 server.go:1019] "Starting client certificate rotation"
Sep 09 20:11:05.971929 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.963620    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:05.971929 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.963652    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:11:05.971929 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.966628    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:05.971929 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.968101    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:11:05.985738 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.985722    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:11:05.989152 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.989119    2579 log.go:25] "Validated CRI v1 image API"
Sep 09 20:11:05.990036 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.990020    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:11:05.993459 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.993441    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:05.994543 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.994526    2579 fs.go:135] Filesystem UUIDs: map[6f040f5d-36e4-4bf9-9b68-69c025fe7f41:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e99df49c-aff3-40f2-afbe-bd3d5c8cc0e5:/dev/nvme0n1p3]
Sep 09 20:11:05.994600 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:05.994543    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:11:06.001021 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.000919    2579 manager.go:216] Machine: {Timestamp:2026-09-09 20:11:05.999457018 +0000 UTC m=+0.375583754 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3182526 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ac075b8afa47b298d8a8e4ff98ba3 SystemUUID:ec2ac075-b8af-a47b-298d-8a8e4ff98ba3 BootID:09fcaef5-3d29-48ae-b474-53a3ced90ecb Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:61:d1:ac:ed:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:61:d1:ac:ed:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:bd:b7:34:a0:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:11:06.001021 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.001016    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:11:06.001146 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.001086    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:11:06.004577 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.004550    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:11:06.004707 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.004581    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-109.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:11:06.005384 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.005373    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:11:06.005415 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.005387    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:11:06.005415 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.005394    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:06.005415 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.005407    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:11:06.006547 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.006538    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:11:06.009837 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.009826    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:06.009956 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.009947    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:11:06.009991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.009982    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:11:06.010017 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.009997    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:11:06.010017 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.010011    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:11:06.010077 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.010020    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:11:06.010077 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.010029    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:11:06.010404 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.010392    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:06.010454 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.010422    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:11:06.012915 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.012899    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:11:06.015907 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.015894    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:11:06.015952 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.015924    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:11:06.017169 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017153    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jj6tw"
Sep 09 20:11:06.017380 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017369    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:11:06.017424 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017386    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:11:06.017424 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017392    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:11:06.017424 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017398    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:11:06.017424 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017409    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:11:06.017424 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017418    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:11:06.017556 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017433    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:11:06.017556 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017438    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:11:06.017556 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017445    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:11:06.017556 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017450    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:11:06.017556 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017458    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:11:06.017556 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.017470    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:11:06.018412 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.018401    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:11:06.018412 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.018413    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:11:06.019233 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.019213    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:11:06.019295 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.019266    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:11:06.022018 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.021999    2579 server.go:1268] "Started kubelet"
Sep 09 20:11:06.022146 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.022110    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:11:06.022262 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.022077    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:11:06.022308 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.022293    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:11:06.022721 ip-10-0-132-109 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:11:06.027368 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.027235    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:11:06.027449 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.027279    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jj6tw"
Sep 09 20:11:06.029314 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.029299    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:11:06.031429 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.031400    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:11:06.031863 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.031849    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-109.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:11:06.032954 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.031860    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-109.ec2.internal.18d3c01346dc4ee4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-109.ec2.internal,UID:ip-10-0-132-109.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-109.ec2.internal,},FirstTimestamp:2026-09-09 20:11:06.021973732 +0000 UTC m=+0.398100468,LastTimestamp:2026-09-09 20:11:06.021973732 +0000 UTC m=+0.398100468,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-109.ec2.internal,}"
Sep 09 20:11:06.035875 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.035855    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:06.036519 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.036490    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:11:06.036613 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.036574    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:06.037247 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037231    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:11:06.037247 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037231    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:11:06.037372 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037257    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:11:06.037372 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037291    2579 factory.go:55] Registering systemd factory
Sep 09 20:11:06.037372 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037307    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:11:06.037372 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037335    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:11:06.037372 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037342    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:11:06.037588 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037533    2579 factory.go:153] Registering CRI-O factory
Sep 09 20:11:06.037588 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037545    2579 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:11:06.037588 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037582    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:11:06.037676 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037597    2579 factory.go:103] Registering Raw factory
Sep 09 20:11:06.037676 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037607    2579 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:11:06.037967 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.037957    2579 manager.go:349] Starting recovery of all containers
Sep 09 20:11:06.038834 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.038817    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:06.041663 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.041637    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-109.ec2.internal\" not found" node="ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.046884 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.046861    2579 manager.go:354] Recovery completed
Sep 09 20:11:06.050503 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.050492    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:06.055763 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.055746    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:06.055830 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.055775    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:06.055830 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.055784    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:06.056369 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.056347    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:11:06.056369 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.056366    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:06.056369 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.056368    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:11:06.056569 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.056404    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:11:06.057049 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.057037    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:06.057096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.057060    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:06.057096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.057069    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:06.057529 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.057517    2579 policy_none.go:49] "None policy: Start"
Sep 09 20:11:06.057578 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.057536    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:11:06.057578 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.057550    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:11:06.058567 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.058553    2579 policy_none.go:47] "Start"
Sep 09 20:11:06.090677 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.090662    2579 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:11:06.090768 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.090696    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:11:06.090768 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.090705    2579 server.go:87] "Starting device plugin registration server"
Sep 09 20:11:06.091009 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.090995    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:11:06.091091 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.091011    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:11:06.091260 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.091242    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:11:06.091354 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.091327    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:11:06.091354 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.091337    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:11:06.093398 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.093261    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:11:06.093398 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.093315    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:06.109973 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.109952    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:11:06.111246 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.111224    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:11:06.111323 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.111278    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:11:06.111323 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.111302    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:11:06.111410 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.111346    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:11:06.113469 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.113448    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:06.191568 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.191502    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:06.192365 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.192344    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:06.192478 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.192383    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:06.192478 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.192417    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:06.192478 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.192444    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.200223 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.200208    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.200273 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.200229    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-109.ec2.internal\": node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:06.211997 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.211976    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-109.ec2.internal"]
Sep 09 20:11:06.212074 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.212039    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:06.215496 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.215480    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:06.215562 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.215506    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:06.215562 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.215516    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:06.216365 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.216342    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:06.218618 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.218606    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:06.218742 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.218726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.218779 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.218759    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:06.219789 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.219766    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:06.219887 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.219796    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:06.219887 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.219770    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:06.219887 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.219809    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:06.219887 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.219818    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:06.219887 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.219828    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:06.220896 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.220879    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.220967 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.220908    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:11:06.222704 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.222688    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:11:06.222756 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.222716    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:11:06.222756 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.222728    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:11:06.237463 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.237442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c35276a998597ed74e89c0d528ad80ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal\" (UID: \"c35276a998597ed74e89c0d528ad80ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.237549 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.237465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c35276a998597ed74e89c0d528ad80ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal\" (UID: \"c35276a998597ed74e89c0d528ad80ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.237549 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.237481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a72042445d935c31fd1e562dad373422-config\") pod \"kube-apiserver-proxy-ip-10-0-132-109.ec2.internal\" (UID: \"a72042445d935c31fd1e562dad373422\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.259807 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.259786    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-109.ec2.internal\" not found" node="ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.264029 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.264014    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-109.ec2.internal\" not found" node="ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.317239 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.317213    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:06.338280 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.338260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a72042445d935c31fd1e562dad373422-config\") pod \"kube-apiserver-proxy-ip-10-0-132-109.ec2.internal\" (UID: \"a72042445d935c31fd1e562dad373422\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.338356 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.338283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c35276a998597ed74e89c0d528ad80ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal\" (UID: \"c35276a998597ed74e89c0d528ad80ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.338356 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.338300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c35276a998597ed74e89c0d528ad80ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal\" (UID: \"c35276a998597ed74e89c0d528ad80ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.338356 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.338306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a72042445d935c31fd1e562dad373422-config\") pod \"kube-apiserver-proxy-ip-10-0-132-109.ec2.internal\" (UID: \"a72042445d935c31fd1e562dad373422\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.338356 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.338333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c35276a998597ed74e89c0d528ad80ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal\" (UID: \"c35276a998597ed74e89c0d528ad80ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.338466 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.338362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c35276a998597ed74e89c0d528ad80ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal\" (UID: \"c35276a998597ed74e89c0d528ad80ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.417477 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.417436    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:06.518201 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.518105    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:06.562534 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.562514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.568862 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.568846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:06.618843 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.618820    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:06.719376 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.719348    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:06.819971 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.819880    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:06.920586 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:06.920546    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:06.963020 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.962996    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:11:06.963472 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.963175    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:06.963472 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:06.963189    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:11:07.021074 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:07.021037    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:07.022970 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:07.022954    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:07.034178 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:07.034149    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:06:06 +0000 UTC" deadline="2028-06-26 11:16:39.009981246 +0000 UTC"
Sep 09 20:11:07.034178 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:07.034173    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15735h5m31.975812248s"
Sep 09 20:11:07.121737 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:07.121711    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:07.144913 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:07.144885    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35276a998597ed74e89c0d528ad80ab.slice/crio-3b1f84998820f6cfb3a8e5fca6d0da91f1c3746c347dea1276c6052b2549f7dd WatchSource:0}: Error finding container 3b1f84998820f6cfb3a8e5fca6d0da91f1c3746c347dea1276c6052b2549f7dd: Status 404 returned error can't find the container with id 3b1f84998820f6cfb3a8e5fca6d0da91f1c3746c347dea1276c6052b2549f7dd
Sep 09 20:11:07.145313 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:07.145292    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72042445d935c31fd1e562dad373422.slice/crio-84e8f174178c59245d97bebb7bee81bc94431dae6cca80d5c4259d647cad2722 WatchSource:0}: Error finding container 84e8f174178c59245d97bebb7bee81bc94431dae6cca80d5c4259d647cad2722: Status 404 returned error can't find the container with id 84e8f174178c59245d97bebb7bee81bc94431dae6cca80d5c4259d647cad2722
Sep 09 20:11:07.151660 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:07.151641    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:11:07.222701 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:07.222663    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:07.323185 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:07.323162    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:07.423697 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:07.423625    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-109.ec2.internal\" not found"
Sep 09 20:11:07.434703 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:07.434682    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:07.437533 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:07.437517    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:07.447726 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:07.447711    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:07.448896 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:07.448884    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-109.ec2.internal"
Sep 09 20:11:07.456776 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:07.456750    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:11:08.000147 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.000097    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:08.011048 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.011025    2579 apiserver.go:52] "Watching apiserver"
Sep 09 20:11:08.020023 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.019998    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:11:08.020526 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.020503    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5l4sk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98","openshift-image-registry/node-ca-8mqdk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal","openshift-multus/multus-additional-cni-plugins-fd985","openshift-multus/multus-nmhdg","kube-system/konnectivity-agent-4lspj","kube-system/kube-apiserver-proxy-ip-10-0-132-109.ec2.internal","openshift-cluster-node-tuning-operator/tuned-8jbnx","openshift-dns/node-resolver-f5dz5","openshift-multus/network-metrics-daemon-v2fjq","openshift-network-diagnostics/network-check-target-7clz9","openshift-network-operator/iptables-alerter-ch7rb","openshift-ovn-kubernetes/ovnkube-node-rk4kx"]
Sep 09 20:11:08.025432 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.025400    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f5dz5"
Sep 09 20:11:08.027072 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.027052    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ch7rb"
Sep 09 20:11:08.027649 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.027630    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.027965 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.027944    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tbs67\""
Sep 09 20:11:08.028044 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.027967    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.028530 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.028511    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8mqdk"
Sep 09 20:11:08.029608 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.029460    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wlx9f\""
Sep 09 20:11:08.029608 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.029472    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.029608 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.029461    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.029608 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.029462    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:11:08.029949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.029933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.030665 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.030428    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.030756 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.030697    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:11:08.030756 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.030730    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.030863 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.030801    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tzbxf\""
Sep 09 20:11:08.031420 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.031399    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.032543 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.032521    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:11:08.032874 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.032851    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.032980 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.032884    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:11:08.032980 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.032904    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:11:08.032980 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.033024    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.033323 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.033305    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:11:08.033423 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.033357    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.033643 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.033625    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.033744 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.033651    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:11:08.033744 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.033694    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:11:08.033863 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.033764    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-87ssb\""
Sep 09 20:11:08.034020 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.034000    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.034110 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.034008    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:11:08.034110 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.034096    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8jqtj\""
Sep 09 20:11:08.035171 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.034986    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.035171 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.035177    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:11:08.035489 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.035471    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-67rzl\""
Sep 09 20:11:08.037062 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.036392    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:11:08.037062 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.036504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:08.037062 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.036920    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.037260 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.037250    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.037337 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.037320    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-45hvc\""
Sep 09 20:11:08.038696 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.038677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.038795 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.038762    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:11:08.039131 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.038926    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:11:08.039340 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.039313    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nkpvf\""
Sep 09 20:11:08.040727 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.040708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:08.040824 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.040775    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:11:08.040824 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.040790    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:08.041438 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.041405    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:11:08.041438 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.041424    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ckcc6\""
Sep 09 20:11:08.042192 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.042167    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:11:08.042484 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.042469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:08.042628 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.042603    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.046246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.046338    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-sysctl-d\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzsw\" (UniqueName: \"kubernetes.io/projected/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-kube-api-access-hbzsw\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af182f8e-b54c-44fe-b03e-69beceef79b9-iptables-alerter-script\") pod \"iptables-alerter-ch7rb\" (UID: \"af182f8e-b54c-44fe-b03e-69beceef79b9\") " pod="openshift-network-operator/iptables-alerter-ch7rb"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-systemd-units\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-log-socket\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-conf-dir\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16-tmp-dir\") pod \"node-resolver-f5dz5\" (UID: \"6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16\") " pod="openshift-dns/node-resolver-f5dz5"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-cni-netd\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-var-lib-cni-multus\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-daemon-config\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-var-lib-kubelet\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-tmp\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.048991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3e83679-e37e-4044-8934-54de8aed342a-konnectivity-ca\") pod \"konnectivity-agent-4lspj\" (UID: \"d3e83679-e37e-4044-8934-54de8aed342a\") " pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-run-ovn\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.049222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f86202c-71cc-4ec5-990b-71f77d5565cf-cni-binary-copy\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-run-systemd\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-run-openvswitch\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb24w\" (UniqueName: \"kubernetes.io/projected/6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16-kube-api-access-gb24w\") pod \"node-resolver-f5dz5\" (UID: \"6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16\") " pod="openshift-dns/node-resolver-f5dz5"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-registration-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-sys-fs\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16-hosts-file\") pod \"node-resolver-f5dz5\" (UID: \"6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16\") " pod="openshift-dns/node-resolver-f5dz5"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmzb9\" (UniqueName: \"kubernetes.io/projected/580818db-cd82-48dc-8ed9-4b7c631e9606-kube-api-access-qmzb9\") pod \"node-ca-8mqdk\" (UID: \"580818db-cd82-48dc-8ed9-4b7c631e9606\") " pod="openshift-image-registry/node-ca-8mqdk"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a49fb02e-529d-46c8-9f79-3f023c94f54a-system-cni-dir\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a49fb02e-529d-46c8-9f79-3f023c94f54a-cnibin\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a49fb02e-529d-46c8-9f79-3f023c94f54a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/580818db-cd82-48dc-8ed9-4b7c631e9606-host\") pod \"node-ca-8mqdk\" (UID: \"580818db-cd82-48dc-8ed9-4b7c631e9606\") " pod="openshift-image-registry/node-ca-8mqdk"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-run\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlhj\" (UniqueName: \"kubernetes.io/projected/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-kube-api-access-qrlhj\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-run-netns\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.049893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-node-log\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-var-lib-kubelet\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc7x2\" (UniqueName: \"kubernetes.io/projected/0f86202c-71cc-4ec5-990b-71f77d5565cf-kube-api-access-hc7x2\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-modprobe-d\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-host\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.049995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-socket-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-kubelet\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqxg6\" (UniqueName: \"kubernetes.io/projected/a49fb02e-529d-46c8-9f79-3f023c94f54a-kube-api-access-gqxg6\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-socket-dir-parent\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-run-multus-certs\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-sysctl-conf\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-device-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/580818db-cd82-48dc-8ed9-4b7c631e9606-serviceca\") pod \"node-ca-8mqdk\" (UID: \"580818db-cd82-48dc-8ed9-4b7c631e9606\") " pod="openshift-image-registry/node-ca-8mqdk"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-etc-openvswitch\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a49fb02e-529d-46c8-9f79-3f023c94f54a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-run-k8s-cni-cncf-io\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-sysconfig\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.050741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050613    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-ovnkube-config\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-env-overrides\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-var-lib-cni-bin\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6t8l\" (UniqueName: \"kubernetes.io/projected/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-kube-api-access-b6t8l\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-cni-dir\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-run-netns\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.050990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-lib-modules\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.051055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3e83679-e37e-4044-8934-54de8aed342a-agent-certs\") pod \"konnectivity-agent-4lspj\" (UID: \"d3e83679-e37e-4044-8934-54de8aed342a\") " pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.051148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-etc-selinux\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.051248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-ovn-node-metrics-cert\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.051290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a49fb02e-529d-46c8-9f79-3f023c94f54a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.051346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-hostroot\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.051402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-sys\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.051457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gfqx\" (UniqueName: \"kubernetes.io/projected/af182f8e-b54c-44fe-b03e-69beceef79b9-kube-api-access-6gfqx\") pod \"iptables-alerter-ch7rb\" (UID: \"af182f8e-b54c-44fe-b03e-69beceef79b9\") " pod="openshift-network-operator/iptables-alerter-ch7rb"
Sep 09 20:11:08.051565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.051495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-ovnkube-script-lib\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.052331 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.051537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-system-cni-dir\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.052788 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.052575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-etc-kubernetes\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.052788 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.052621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-systemd\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.052788 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.052682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-tuned\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.052788 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.052722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-slash\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.052788 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.052750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-cni-bin\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.052788 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.052779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-kubernetes\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.053065 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.052812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-var-lib-openvswitch\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.053065 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.052854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a49fb02e-529d-46c8-9f79-3f023c94f54a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.053065 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.052886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-cnibin\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.053065 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.052907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.053065 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.052924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af182f8e-b54c-44fe-b03e-69beceef79b9-host-slash\") pod \"iptables-alerter-ch7rb\" (UID: \"af182f8e-b54c-44fe-b03e-69beceef79b9\") " pod="openshift-network-operator/iptables-alerter-ch7rb"
Sep 09 20:11:08.073665 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.073638    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kp5mq"
Sep 09 20:11:08.082221 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.082200    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kp5mq"
Sep 09 20:11:08.117693 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.117602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal" event={"ID":"c35276a998597ed74e89c0d528ad80ab","Type":"ContainerStarted","Data":"3b1f84998820f6cfb3a8e5fca6d0da91f1c3746c347dea1276c6052b2549f7dd"}
Sep 09 20:11:08.119971 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.119786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-109.ec2.internal" event={"ID":"a72042445d935c31fd1e562dad373422","Type":"ContainerStarted","Data":"84e8f174178c59245d97bebb7bee81bc94431dae6cca80d5c4259d647cad2722"}
Sep 09 20:11:08.138644 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.138556    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:11:08.153547 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-var-lib-kubelet\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.153668 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc7x2\" (UniqueName: \"kubernetes.io/projected/0f86202c-71cc-4ec5-990b-71f77d5565cf-kube-api-access-hc7x2\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.153668 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-modprobe-d\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.153668 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-host\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.153668 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-socket-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.153668 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-kubelet\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqxg6\" (UniqueName: \"kubernetes.io/projected/a49fb02e-529d-46c8-9f79-3f023c94f54a-kube-api-access-gqxg6\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-socket-dir-parent\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-run-multus-certs\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-sysctl-conf\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-device-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/580818db-cd82-48dc-8ed9-4b7c631e9606-serviceca\") pod \"node-ca-8mqdk\" (UID: \"580818db-cd82-48dc-8ed9-4b7c631e9606\") " pod="openshift-image-registry/node-ca-8mqdk"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-etc-openvswitch\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a49fb02e-529d-46c8-9f79-3f023c94f54a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-run-k8s-cni-cncf-io\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-sysconfig\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:08.153949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-ovnkube-config\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-env-overrides\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.153992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-var-lib-cni-bin\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6t8l\" (UniqueName: \"kubernetes.io/projected/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-kube-api-access-b6t8l\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-cni-dir\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-run-netns\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-lib-modules\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3e83679-e37e-4044-8934-54de8aed342a-agent-certs\") pod \"konnectivity-agent-4lspj\" (UID: \"d3e83679-e37e-4044-8934-54de8aed342a\") " pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a213458e-bfea-4d69-aa91-e3a039bfa810-kubelet-config\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-run-netns\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-kubelet\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154298    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-host\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-modprobe-d\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-var-lib-kubelet\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-sysconfig\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-var-lib-cni-bin\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.154667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-socket-dir-parent\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-etc-openvswitch\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-run-multus-certs\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-cni-dir\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-lib-modules\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-sysctl-conf\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154734    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-device-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.154893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-env-overrides\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-ovnkube-config\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155152    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-run-k8s-cni-cncf-io\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-etc-selinux\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-ovn-node-metrics-cert\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/580818db-cd82-48dc-8ed9-4b7c631e9606-serviceca\") pod \"node-ca-8mqdk\" (UID: \"580818db-cd82-48dc-8ed9-4b7c631e9606\") " pod="openshift-image-registry/node-ca-8mqdk"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a49fb02e-529d-46c8-9f79-3f023c94f54a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-hostroot\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155334    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:11:08.155506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-sys\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gfqx\" (UniqueName: \"kubernetes.io/projected/af182f8e-b54c-44fe-b03e-69beceef79b9-kube-api-access-6gfqx\") pod \"iptables-alerter-ch7rb\" (UID: \"af182f8e-b54c-44fe-b03e-69beceef79b9\") " pod="openshift-network-operator/iptables-alerter-ch7rb"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-ovnkube-script-lib\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-system-cni-dir\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-etc-kubernetes\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-systemd\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-tuned\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-slash\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-cni-bin\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-kubernetes\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcs7h\" (UniqueName: \"kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h\") pod \"network-check-target-7clz9\" (UID: \"16520da4-678a-4a74-bb1e-6931c60e0f75\") " pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-var-lib-openvswitch\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a49fb02e-529d-46c8-9f79-3f023c94f54a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-cnibin\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-system-cni-dir\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a49fb02e-529d-46c8-9f79-3f023c94f54a-cni-binary-copy\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:08.156284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-etc-kubernetes\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af182f8e-b54c-44fe-b03e-69beceef79b9-host-slash\") pod \"iptables-alerter-ch7rb\" (UID: \"af182f8e-b54c-44fe-b03e-69beceef79b9\") " pod="openshift-network-operator/iptables-alerter-ch7rb"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-systemd\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-hostroot\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-socket-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-var-lib-openvswitch\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-cni-bin\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-slash\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-sysctl-d\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzsw\" (UniqueName: \"kubernetes.io/projected/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-kube-api-access-hbzsw\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af182f8e-b54c-44fe-b03e-69beceef79b9-iptables-alerter-script\") pod \"iptables-alerter-ch7rb\" (UID: \"af182f8e-b54c-44fe-b03e-69beceef79b9\") " pod="openshift-network-operator/iptables-alerter-ch7rb"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-systemd-units\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-log-socket\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155816    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-sys\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-cnibin\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.156991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-ovnkube-script-lib\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a49fb02e-529d-46c8-9f79-3f023c94f54a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-sysctl-d\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-kubernetes\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-etc-selinux\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-systemd-units\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-log-socket\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.155378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a49fb02e-529d-46c8-9f79-3f023c94f54a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-conf-dir\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156768    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af182f8e-b54c-44fe-b03e-69beceef79b9-host-slash\") pod \"iptables-alerter-ch7rb\" (UID: \"af182f8e-b54c-44fe-b03e-69beceef79b9\") " pod="openshift-network-operator/iptables-alerter-ch7rb"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/af182f8e-b54c-44fe-b03e-69beceef79b9-iptables-alerter-script\") pod \"iptables-alerter-ch7rb\" (UID: \"af182f8e-b54c-44fe-b03e-69beceef79b9\") " pod="openshift-network-operator/iptables-alerter-ch7rb"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-conf-dir\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a213458e-bfea-4d69-aa91-e3a039bfa810-dbus\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16-tmp-dir\") pod \"node-resolver-f5dz5\" (UID: \"6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16\") " pod="openshift-dns/node-resolver-f5dz5"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-cni-netd\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.156980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-var-lib-cni-multus\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-daemon-config\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.157772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-var-lib-kubelet\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-tmp\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3e83679-e37e-4044-8934-54de8aed342a-konnectivity-ca\") pod \"konnectivity-agent-4lspj\" (UID: \"d3e83679-e37e-4044-8934-54de8aed342a\") " pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-run-ovn\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f86202c-71cc-4ec5-990b-71f77d5565cf-cni-binary-copy\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-run-systemd\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-run-openvswitch\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb24w\" (UniqueName: \"kubernetes.io/projected/6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16-kube-api-access-gb24w\") pod \"node-resolver-f5dz5\" (UID: \"6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16\") " pod="openshift-dns/node-resolver-f5dz5"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-registration-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-sys-fs\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16-hosts-file\") pod \"node-resolver-f5dz5\" (UID: \"6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16\") " pod="openshift-dns/node-resolver-f5dz5"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzb9\" (UniqueName: \"kubernetes.io/projected/580818db-cd82-48dc-8ed9-4b7c631e9606-kube-api-access-qmzb9\") pod \"node-ca-8mqdk\" (UID: \"580818db-cd82-48dc-8ed9-4b7c631e9606\") " pod="openshift-image-registry/node-ca-8mqdk"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a49fb02e-529d-46c8-9f79-3f023c94f54a-system-cni-dir\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a49fb02e-529d-46c8-9f79-3f023c94f54a-cnibin\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a49fb02e-529d-46c8-9f79-3f023c94f54a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/580818db-cd82-48dc-8ed9-4b7c631e9606-host\") pod \"node-ca-8mqdk\" (UID: \"580818db-cd82-48dc-8ed9-4b7c631e9606\") " pod="openshift-image-registry/node-ca-8mqdk"
Sep 09 20:11:08.158731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d3e83679-e37e-4044-8934-54de8aed342a-konnectivity-ca\") pod \"konnectivity-agent-4lspj\" (UID: \"d3e83679-e37e-4044-8934-54de8aed342a\") " pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-run\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-run-systemd\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f86202c-71cc-4ec5-990b-71f77d5565cf-host-var-lib-cni-multus\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24xck\" (UniqueName: \"kubernetes.io/projected/cd1b99c8-4157-4934-b9e3-1b7f51835443-kube-api-access-24xck\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f86202c-71cc-4ec5-990b-71f77d5565cf-cni-binary-copy\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-var-lib-kubelet\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16-hosts-file\") pod \"node-resolver-f5dz5\" (UID: \"6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16\") " pod="openshift-dns/node-resolver-f5dz5"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlhj\" (UniqueName: \"kubernetes.io/projected/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-kube-api-access-qrlhj\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-run-netns\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a49fb02e-529d-46c8-9f79-3f023c94f54a-cnibin\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16-tmp-dir\") pod \"node-resolver-f5dz5\" (UID: \"6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16\") " pod="openshift-dns/node-resolver-f5dz5"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-node-log\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-sys-fs\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-cni-netd\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a49fb02e-529d-46c8-9f79-3f023c94f54a-system-cni-dir\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-registration-dir\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.159587 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-run-openvswitch\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.160481 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.157881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-run-ovn\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.160481 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.158680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f86202c-71cc-4ec5-990b-71f77d5565cf-multus-daemon-config\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.160481 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.159169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-run-netns\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.160481 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.159179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-run\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.160481 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.159215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-node-log\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.160481 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.159322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.160481 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.159379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/580818db-cd82-48dc-8ed9-4b7c631e9606-host\") pod \"node-ca-8mqdk\" (UID: \"580818db-cd82-48dc-8ed9-4b7c631e9606\") " pod="openshift-image-registry/node-ca-8mqdk"
Sep 09 20:11:08.160481 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.159644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a49fb02e-529d-46c8-9f79-3f023c94f54a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.162476 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.162454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-tmp\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.163285 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.162492    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-etc-tuned\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.164016 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.163958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d3e83679-e37e-4044-8934-54de8aed342a-agent-certs\") pod \"konnectivity-agent-4lspj\" (UID: \"d3e83679-e37e-4044-8934-54de8aed342a\") " pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:08.180611 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.176868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-ovn-node-metrics-cert\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.180611 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.177354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzsw\" (UniqueName: \"kubernetes.io/projected/8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a-kube-api-access-hbzsw\") pod \"tuned-8jbnx\" (UID: \"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a\") " pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.180611 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.177959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6t8l\" (UniqueName: \"kubernetes.io/projected/8f9801ea-4417-4cd9-9c0f-afde235d5fe5-kube-api-access-b6t8l\") pod \"ovnkube-node-rk4kx\" (UID: \"8f9801ea-4417-4cd9-9c0f-afde235d5fe5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.180611 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.178429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc7x2\" (UniqueName: \"kubernetes.io/projected/0f86202c-71cc-4ec5-990b-71f77d5565cf-kube-api-access-hc7x2\") pod \"multus-nmhdg\" (UID: \"0f86202c-71cc-4ec5-990b-71f77d5565cf\") " pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.180611 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.179152    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmzb9\" (UniqueName: \"kubernetes.io/projected/580818db-cd82-48dc-8ed9-4b7c631e9606-kube-api-access-qmzb9\") pod \"node-ca-8mqdk\" (UID: \"580818db-cd82-48dc-8ed9-4b7c631e9606\") " pod="openshift-image-registry/node-ca-8mqdk"
Sep 09 20:11:08.180611 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.179673    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlhj\" (UniqueName: \"kubernetes.io/projected/1f392cbd-f20d-44e0-8f2f-4f0687cc1f87-kube-api-access-qrlhj\") pod \"aws-ebs-csi-driver-node-t5g98\" (UID: \"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.180611 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.180122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gfqx\" (UniqueName: \"kubernetes.io/projected/af182f8e-b54c-44fe-b03e-69beceef79b9-kube-api-access-6gfqx\") pod \"iptables-alerter-ch7rb\" (UID: \"af182f8e-b54c-44fe-b03e-69beceef79b9\") " pod="openshift-network-operator/iptables-alerter-ch7rb"
Sep 09 20:11:08.180611 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.180567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb24w\" (UniqueName: \"kubernetes.io/projected/6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16-kube-api-access-gb24w\") pod \"node-resolver-f5dz5\" (UID: \"6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16\") " pod="openshift-dns/node-resolver-f5dz5"
Sep 09 20:11:08.182904 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.181484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqxg6\" (UniqueName: \"kubernetes.io/projected/a49fb02e-529d-46c8-9f79-3f023c94f54a-kube-api-access-gqxg6\") pod \"multus-additional-cni-plugins-fd985\" (UID: \"a49fb02e-529d-46c8-9f79-3f023c94f54a\") " pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.191338 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.191291    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:11:08.259804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.259714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:08.259804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.259766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a213458e-bfea-4d69-aa91-e3a039bfa810-kubelet-config\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:08.259804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.259801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcs7h\" (UniqueName: \"kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h\") pod \"network-check-target-7clz9\" (UID: \"16520da4-678a-4a74-bb1e-6931c60e0f75\") " pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:08.260096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.259827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:08.260096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.259855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a213458e-bfea-4d69-aa91-e3a039bfa810-dbus\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:08.260096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.259897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24xck\" (UniqueName: \"kubernetes.io/projected/cd1b99c8-4157-4934-b9e3-1b7f51835443-kube-api-access-24xck\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:08.260310 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.260273    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:08.260366 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.260346    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret podName:a213458e-bfea-4d69-aa91-e3a039bfa810 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:08.760323834 +0000 UTC m=+3.136450564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret") pod "global-pull-secret-syncer-5l4sk" (UID: "a213458e-bfea-4d69-aa91-e3a039bfa810") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:08.260586 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.260553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a213458e-bfea-4d69-aa91-e3a039bfa810-kubelet-config\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:08.260829 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.260809    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:08.260905 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.260864    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs podName:cd1b99c8-4157-4934-b9e3-1b7f51835443 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:08.760849816 +0000 UTC m=+3.136976542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs") pod "network-metrics-daemon-v2fjq" (UID: "cd1b99c8-4157-4934-b9e3-1b7f51835443") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:08.260970 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.260918    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a213458e-bfea-4d69-aa91-e3a039bfa810-dbus\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:08.266973 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.266815    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:08.266973 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.266838    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:08.266973 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.266853    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lcs7h for pod openshift-network-diagnostics/network-check-target-7clz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:08.266973 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.266921    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h podName:16520da4-678a-4a74-bb1e-6931c60e0f75 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:08.766904601 +0000 UTC m=+3.143031331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lcs7h" (UniqueName: "kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h") pod "network-check-target-7clz9" (UID: "16520da4-678a-4a74-bb1e-6931c60e0f75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:08.269830 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.269780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24xck\" (UniqueName: \"kubernetes.io/projected/cd1b99c8-4157-4934-b9e3-1b7f51835443-kube-api-access-24xck\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:08.339800 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.339764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f5dz5"
Sep 09 20:11:08.351050 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.351018    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ch7rb"
Sep 09 20:11:08.363142 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.363116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8mqdk"
Sep 09 20:11:08.367604 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.367573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:08.375133 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.375113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fd985"
Sep 09 20:11:08.385301 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.385159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nmhdg"
Sep 09 20:11:08.392272 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.392093    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8jbnx"
Sep 09 20:11:08.399259 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.399238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:08.405105 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.405082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98"
Sep 09 20:11:08.764865 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.764826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:08.765042 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.764903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:08.765042 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.765013    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:08.765203 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.765091    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret podName:a213458e-bfea-4d69-aa91-e3a039bfa810 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:09.765070157 +0000 UTC m=+4.141196893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret") pod "global-pull-secret-syncer-5l4sk" (UID: "a213458e-bfea-4d69-aa91-e3a039bfa810") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:08.765203 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.765016    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:08.765203 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.765183    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs podName:cd1b99c8-4157-4934-b9e3-1b7f51835443 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:09.765162983 +0000 UTC m=+4.141289712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs") pod "network-metrics-daemon-v2fjq" (UID: "cd1b99c8-4157-4934-b9e3-1b7f51835443") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:08.865983 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:08.865945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcs7h\" (UniqueName: \"kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h\") pod \"network-check-target-7clz9\" (UID: \"16520da4-678a-4a74-bb1e-6931c60e0f75\") " pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:08.866185 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.866098    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:08.866185 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.866119    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:08.866185 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.866132    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lcs7h for pod openshift-network-diagnostics/network-check-target-7clz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:08.866392 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:08.866200    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h podName:16520da4-678a-4a74-bb1e-6931c60e0f75 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:09.866185985 +0000 UTC m=+4.242312710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcs7h" (UniqueName: "kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h") pod "network-check-target-7clz9" (UID: "16520da4-678a-4a74-bb1e-6931c60e0f75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:09.084944 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:09.084837    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:08 +0000 UTC" deadline="2028-02-29 20:39:32.908305092 +0000 UTC"
Sep 09 20:11:09.084944 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:09.084875    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12912h28m23.823434267s"
Sep 09 20:11:09.112586 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:09.112559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:09.112740 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:09.112679    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:09.284396 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:09.284348    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf182f8e_b54c_44fe_b03e_69beceef79b9.slice/crio-54aa42c6220c52d31ec34d4c22eda684f8972cb3dbd510dfb108de101eba16ff WatchSource:0}: Error finding container 54aa42c6220c52d31ec34d4c22eda684f8972cb3dbd510dfb108de101eba16ff: Status 404 returned error can't find the container with id 54aa42c6220c52d31ec34d4c22eda684f8972cb3dbd510dfb108de101eba16ff
Sep 09 20:11:09.288857 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:09.288830    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f86202c_71cc_4ec5_990b_71f77d5565cf.slice/crio-bc84d35b356e9f8923aaaec7ac20564fb92e2a146f6adfd3c0e9a90cae417812 WatchSource:0}: Error finding container bc84d35b356e9f8923aaaec7ac20564fb92e2a146f6adfd3c0e9a90cae417812: Status 404 returned error can't find the container with id bc84d35b356e9f8923aaaec7ac20564fb92e2a146f6adfd3c0e9a90cae417812
Sep 09 20:11:09.772576 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:09.772534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:09.772749 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:09.772624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:09.772813 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:09.772765    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:09.772865 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:09.772827    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs podName:cd1b99c8-4157-4934-b9e3-1b7f51835443 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:11.772808076 +0000 UTC m=+6.148934816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs") pod "network-metrics-daemon-v2fjq" (UID: "cd1b99c8-4157-4934-b9e3-1b7f51835443") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:09.773294 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:09.773274    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:09.773377 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:09.773333    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret podName:a213458e-bfea-4d69-aa91-e3a039bfa810 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:11.773316602 +0000 UTC m=+6.149443343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret") pod "global-pull-secret-syncer-5l4sk" (UID: "a213458e-bfea-4d69-aa91-e3a039bfa810") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:09.873466 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:09.873431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcs7h\" (UniqueName: \"kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h\") pod \"network-check-target-7clz9\" (UID: \"16520da4-678a-4a74-bb1e-6931c60e0f75\") " pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:09.873653 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:09.873635    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:09.873700 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:09.873661    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:09.873700 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:09.873675    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lcs7h for pod openshift-network-diagnostics/network-check-target-7clz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:09.873771 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:09.873765    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h podName:16520da4-678a-4a74-bb1e-6931c60e0f75 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:11.87374367 +0000 UTC m=+6.249870413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcs7h" (UniqueName: "kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h") pod "network-check-target-7clz9" (UID: "16520da4-678a-4a74-bb1e-6931c60e0f75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:10.085428 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.085299    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:06:08 +0000 UTC" deadline="2028-02-06 20:44:26.953387761 +0000 UTC"
Sep 09 20:11:10.085428 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.085370    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12360h33m16.868024012s"
Sep 09 20:11:10.112932 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.112300    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:10.112932 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:10.112431    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:10.112932 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.112300    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:10.112932 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:10.112889    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:10.138250 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.138198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-109.ec2.internal" event={"ID":"a72042445d935c31fd1e562dad373422","Type":"ContainerStarted","Data":"71739206b33235b0edf241dfd3e1041ff200923911cca321340a03870c08f811"}
Sep 09 20:11:10.142800 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.142769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8mqdk" event={"ID":"580818db-cd82-48dc-8ed9-4b7c631e9606","Type":"ContainerStarted","Data":"43e88014ce597818c6cf9cfcb7a7426b66bf0139240a58f8aa36adf26e38b70b"}
Sep 09 20:11:10.153793 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.153736    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-109.ec2.internal" podStartSLOduration=2.153719966 podStartE2EDuration="2.153719966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:10.153148789 +0000 UTC m=+4.529275537" watchObservedRunningTime="2026-09-09 20:11:10.153719966 +0000 UTC m=+4.529846716"
Sep 09 20:11:10.165549 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.165520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4lspj" event={"ID":"d3e83679-e37e-4044-8934-54de8aed342a","Type":"ContainerStarted","Data":"121df9aafb87f95eac841ad29fab248e4a6e85eac1b5ee14aa834df4208492b3"}
Sep 09 20:11:10.172764 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.172734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f5dz5" event={"ID":"6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16","Type":"ContainerStarted","Data":"d3967fb1c6a3572015a25e316e76460b64b3e429a1696a5d5cd8a64bd09805b2"}
Sep 09 20:11:10.181163 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.181129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nmhdg" event={"ID":"0f86202c-71cc-4ec5-990b-71f77d5565cf","Type":"ContainerStarted","Data":"bc84d35b356e9f8923aaaec7ac20564fb92e2a146f6adfd3c0e9a90cae417812"}
Sep 09 20:11:10.194639 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.194610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd985" event={"ID":"a49fb02e-529d-46c8-9f79-3f023c94f54a","Type":"ContainerStarted","Data":"435d21db5263560c29f9ef9f666e478678ea0a25ddc1a86058df9ea58808288b"}
Sep 09 20:11:10.200563 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.199675    2579 generic.go:356] "Generic (PLEG): container finished" podID="c35276a998597ed74e89c0d528ad80ab" containerID="b7af5456807abc16c232b653e6392da974217bf663ecba612afc525c67973c6b" exitCode=0
Sep 09 20:11:10.200563 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.199766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal" event={"ID":"c35276a998597ed74e89c0d528ad80ab","Type":"ContainerDied","Data":"b7af5456807abc16c232b653e6392da974217bf663ecba612afc525c67973c6b"}
Sep 09 20:11:10.210960 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.210870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98" event={"ID":"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87","Type":"ContainerStarted","Data":"024d92df7bcf505687d95e2a0bbf2cd83db8cf197cacb4e34397f2e30d070ed4"}
Sep 09 20:11:10.225728 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.225696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8jbnx" event={"ID":"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a","Type":"ContainerStarted","Data":"56c1f86a4619ee4936eff74b6e69a036664f5f408d5691c92742c657793013fc"}
Sep 09 20:11:10.242154 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.242116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx" event={"ID":"8f9801ea-4417-4cd9-9c0f-afde235d5fe5","Type":"ContainerStarted","Data":"f8e5d6b90d43cefd097172011d0be55032cf894a6e838a1f081746a065a1275b"}
Sep 09 20:11:10.245770 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:10.245663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ch7rb" event={"ID":"af182f8e-b54c-44fe-b03e-69beceef79b9","Type":"ContainerStarted","Data":"54aa42c6220c52d31ec34d4c22eda684f8972cb3dbd510dfb108de101eba16ff"}
Sep 09 20:11:11.112194 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:11.112158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:11.112680 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:11.112304    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:11.257297 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:11.257262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal" event={"ID":"c35276a998597ed74e89c0d528ad80ab","Type":"ContainerStarted","Data":"1884ccee34d9f08817b691545c1bab917dd58bb496627f37d00056c6a5323ddc"}
Sep 09 20:11:11.794403 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:11.793529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:11.794403 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:11.793614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:11.794403 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:11.793757    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:11.794403 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:11.793831    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs podName:cd1b99c8-4157-4934-b9e3-1b7f51835443 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:15.793808265 +0000 UTC m=+10.169935006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs") pod "network-metrics-daemon-v2fjq" (UID: "cd1b99c8-4157-4934-b9e3-1b7f51835443") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:11.794403 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:11.794274    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:11.794403 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:11.794334    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret podName:a213458e-bfea-4d69-aa91-e3a039bfa810 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:15.794316515 +0000 UTC m=+10.170443256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret") pod "global-pull-secret-syncer-5l4sk" (UID: "a213458e-bfea-4d69-aa91-e3a039bfa810") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:11.894789 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:11.894707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcs7h\" (UniqueName: \"kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h\") pod \"network-check-target-7clz9\" (UID: \"16520da4-678a-4a74-bb1e-6931c60e0f75\") " pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:11.894964 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:11.894883    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:11.894964 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:11.894905    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:11.894964 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:11.894918    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lcs7h for pod openshift-network-diagnostics/network-check-target-7clz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:11.895109 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:11.894984    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h podName:16520da4-678a-4a74-bb1e-6931c60e0f75 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:15.894962161 +0000 UTC m=+10.271088900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcs7h" (UniqueName: "kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h") pod "network-check-target-7clz9" (UID: "16520da4-678a-4a74-bb1e-6931c60e0f75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:12.115083 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:12.115045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:12.115589 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:12.115244    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:12.115673 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:12.115657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:12.115799 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:12.115770    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:13.112453 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:13.112414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:13.112651 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:13.112530    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:14.112963 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:14.112697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:14.113466 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:14.112827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:14.113466 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:14.113057    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:14.113466 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:14.113180    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:15.112461 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:15.112389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:15.112699 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:15.112594    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:15.829390 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:15.829339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:15.829835 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:15.829421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:15.829835 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:15.829502    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:15.829835 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:15.829538    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:15.829835 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:15.829580    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret podName:a213458e-bfea-4d69-aa91-e3a039bfa810 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:23.82955852 +0000 UTC m=+18.205685275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret") pod "global-pull-secret-syncer-5l4sk" (UID: "a213458e-bfea-4d69-aa91-e3a039bfa810") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:15.829835 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:15.829603    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs podName:cd1b99c8-4157-4934-b9e3-1b7f51835443 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:23.829591529 +0000 UTC m=+18.205718255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs") pod "network-metrics-daemon-v2fjq" (UID: "cd1b99c8-4157-4934-b9e3-1b7f51835443") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:15.930439 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:15.930399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcs7h\" (UniqueName: \"kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h\") pod \"network-check-target-7clz9\" (UID: \"16520da4-678a-4a74-bb1e-6931c60e0f75\") " pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:15.930669 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:15.930646    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:15.930669 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:15.930671    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:15.930864 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:15.930684    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lcs7h for pod openshift-network-diagnostics/network-check-target-7clz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:15.930864 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:15.930812    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h podName:16520da4-678a-4a74-bb1e-6931c60e0f75 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:23.930790404 +0000 UTC m=+18.306917143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcs7h" (UniqueName: "kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h") pod "network-check-target-7clz9" (UID: "16520da4-678a-4a74-bb1e-6931c60e0f75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:16.115504 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:16.114491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:16.115504 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:16.114624    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:16.115504 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:16.115008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:16.115504 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:16.115095    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:17.111799 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:17.111761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:17.112347 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:17.111893    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:18.114267 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:18.114234    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:18.114702 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:18.114360    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:18.114702 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:18.114407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:18.114702 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:18.114519    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:19.112028 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:19.111997    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:19.112207 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:19.112142    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:20.114355 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:20.114331    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:20.114702 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:20.114418    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:20.114702 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:20.114495    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:20.114702 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:20.114580    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:21.111695 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:21.111663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:21.111883 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:21.111771    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:22.111801 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:22.111764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:22.112227 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:22.111780    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:22.112227 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:22.111880    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:22.112227 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:22.111991    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:23.112552 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:23.112519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:23.113055 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:23.112628    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:23.890385 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:23.890346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:23.890565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:23.890425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:23.890565 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:23.890491    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:23.890565 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:23.890520    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:23.890565 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:23.890558    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret podName:a213458e-bfea-4d69-aa91-e3a039bfa810 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:39.890540225 +0000 UTC m=+34.266666960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret") pod "global-pull-secret-syncer-5l4sk" (UID: "a213458e-bfea-4d69-aa91-e3a039bfa810") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:23.890730 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:23.890572    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs podName:cd1b99c8-4157-4934-b9e3-1b7f51835443 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:39.890565777 +0000 UTC m=+34.266692500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs") pod "network-metrics-daemon-v2fjq" (UID: "cd1b99c8-4157-4934-b9e3-1b7f51835443") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:23.991616 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:23.991590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcs7h\" (UniqueName: \"kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h\") pod \"network-check-target-7clz9\" (UID: \"16520da4-678a-4a74-bb1e-6931c60e0f75\") " pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:23.991766 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:23.991712    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:23.991766 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:23.991727    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:23.991766 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:23.991735    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lcs7h for pod openshift-network-diagnostics/network-check-target-7clz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:23.991890 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:23.991785    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h podName:16520da4-678a-4a74-bb1e-6931c60e0f75 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:39.99176954 +0000 UTC m=+34.367896264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcs7h" (UniqueName: "kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h") pod "network-check-target-7clz9" (UID: "16520da4-678a-4a74-bb1e-6931c60e0f75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:24.111791 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:24.111767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:24.111975 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:24.111878    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:24.111975 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:24.111962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:24.112102 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:24.112078    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:25.111836 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:25.111801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:25.112250 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:25.111916    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:26.112728 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:26.112231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:26.112728 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:26.112356    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:26.112728 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:26.112427    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:26.112728 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:26.112561    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:27.111927 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:27.111885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:27.112081 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:27.112013    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:28.112427 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.112235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:28.113002 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.112237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:28.113002 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:28.112508    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:28.113002 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:28.112593    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:28.294372 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.294292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98" event={"ID":"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87","Type":"ContainerStarted","Data":"61189228c5fbcfe46fad63743a2dc38c74723d6c6de5c0c4374cb6b9b7560abf"}
Sep 09 20:11:28.296497 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.295991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8jbnx" event={"ID":"8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a","Type":"ContainerStarted","Data":"7db10f05e0636e28b5297c65917b7adef9db68b97b8bb8f0854e27930a2b6792"}
Sep 09 20:11:28.299701 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.299677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx" event={"ID":"8f9801ea-4417-4cd9-9c0f-afde235d5fe5","Type":"ContainerStarted","Data":"6a83ddf25c61302248ad633b7b632fee1e4d953c90ce3712a40e72da78ebea5c"}
Sep 09 20:11:28.299823 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.299749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx" event={"ID":"8f9801ea-4417-4cd9-9c0f-afde235d5fe5","Type":"ContainerStarted","Data":"bd2840251c9f5d9cf6efde5d2c58b2e2be5afcc3c746ff5898f2d8a5e1b6c823"}
Sep 09 20:11:28.299823 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.299803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx" event={"ID":"8f9801ea-4417-4cd9-9c0f-afde235d5fe5","Type":"ContainerStarted","Data":"25a7f68b9bfb38e067a9c12b36d27461ef215e485ea869a186b045060e1419e6"}
Sep 09 20:11:28.299941 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.299851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx" event={"ID":"8f9801ea-4417-4cd9-9c0f-afde235d5fe5","Type":"ContainerStarted","Data":"3c61eb9cc0cd49fede861136edd4d77df83efd76a41686a8c59df205285e8a62"}
Sep 09 20:11:28.299941 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.299899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx" event={"ID":"8f9801ea-4417-4cd9-9c0f-afde235d5fe5","Type":"ContainerStarted","Data":"2d3956a2214ac5d9735704861044d95792a3313545d33e43bdcdb8e2a093fc03"}
Sep 09 20:11:28.299941 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.299938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx" event={"ID":"8f9801ea-4417-4cd9-9c0f-afde235d5fe5","Type":"ContainerStarted","Data":"d0b4f74ef3f75438094873a3a4e111963364e2a84b7dc0e8f5a8af08f20171a1"}
Sep 09 20:11:28.301282 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.301253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8mqdk" event={"ID":"580818db-cd82-48dc-8ed9-4b7c631e9606","Type":"ContainerStarted","Data":"bc1f899de32cc03aab63af7400c81827c5a363227aab188a489e74dfc23497b4"}
Sep 09 20:11:28.303022 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.302993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4lspj" event={"ID":"d3e83679-e37e-4044-8934-54de8aed342a","Type":"ContainerStarted","Data":"55f06be3fd0dd4d0b510e21e86db2454ad23309449d817053aa59d8437a25fab"}
Sep 09 20:11:28.304772 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.304745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f5dz5" event={"ID":"6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16","Type":"ContainerStarted","Data":"35244f929e269fcbbf2f734af2d7891e797beb277bc025738ee846f20b7622ec"}
Sep 09 20:11:28.306669 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.306647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nmhdg" event={"ID":"0f86202c-71cc-4ec5-990b-71f77d5565cf","Type":"ContainerStarted","Data":"f68d66c2bf3af8e1896965fc395f9e96b3e7738e569a67c20bd8d5dd72988894"}
Sep 09 20:11:28.308479 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.308455    2579 generic.go:356] "Generic (PLEG): container finished" podID="a49fb02e-529d-46c8-9f79-3f023c94f54a" containerID="a673b46b79871d9912c201fa05490866d0756761acf7f4421b17fe5593f66a69" exitCode=0
Sep 09 20:11:28.308570 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.308488    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd985" event={"ID":"a49fb02e-529d-46c8-9f79-3f023c94f54a","Type":"ContainerDied","Data":"a673b46b79871d9912c201fa05490866d0756761acf7f4421b17fe5593f66a69"}
Sep 09 20:11:28.314737 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.314696    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-109.ec2.internal" podStartSLOduration=20.314684446 podStartE2EDuration="20.314684446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:11.274685714 +0000 UTC m=+5.650812462" watchObservedRunningTime="2026-09-09 20:11:28.314684446 +0000 UTC m=+22.690811191"
Sep 09 20:11:28.314947 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.314924    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8jbnx" podStartSLOduration=3.101872795 podStartE2EDuration="21.314918271s" totalImagesPullingTime="18.213045476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:28.314473579 +0000 UTC m=+22.690600325" watchObservedRunningTime="2026-09-09 20:11:28.314918271 +0000 UTC m=+22.691045016"
Sep 09 20:11:28.331014 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.330977    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f5dz5" podStartSLOduration=3.116097823 podStartE2EDuration="21.330966284s" totalImagesPullingTime="18.214868461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:28.330740774 +0000 UTC m=+22.706867518" watchObservedRunningTime="2026-09-09 20:11:28.330966284 +0000 UTC m=+22.707093028"
Sep 09 20:11:28.347835 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.347783    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8mqdk" podStartSLOduration=3.136185139 podStartE2EDuration="21.34777411s" totalImagesPullingTime="18.211588971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:28.347321765 +0000 UTC m=+22.723448512" watchObservedRunningTime="2026-09-09 20:11:28.34777411 +0000 UTC m=+22.723900854"
Sep 09 20:11:28.382802 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.382755    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nmhdg" podStartSLOduration=3.116005171 podStartE2EDuration="21.382742076s" totalImagesPullingTime="18.266736905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:28.38245736 +0000 UTC m=+22.758584104" watchObservedRunningTime="2026-09-09 20:11:28.382742076 +0000 UTC m=+22.758868862"
Sep 09 20:11:28.396549 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.396503    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4lspj" podStartSLOduration=10.307320868 podStartE2EDuration="21.39649283s" totalImagesPullingTime="11.089171962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:28.396457246 +0000 UTC m=+22.772583992" watchObservedRunningTime="2026-09-09 20:11:28.39649283 +0000 UTC m=+22.772619576"
Sep 09 20:11:28.994384 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:28.994188    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:11:29.104468 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:29.104251    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:11:28.994379874Z","UUID":"0b784e85-59cf-4792-90fa-634e221d0707","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:11:29.106109 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:29.106089    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:11:29.106240 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:29.106121    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:11:29.111560 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:29.111540    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:29.111680 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:29.111656    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:29.313189 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:29.313086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ch7rb" event={"ID":"af182f8e-b54c-44fe-b03e-69beceef79b9","Type":"ContainerStarted","Data":"698a79b1d921da9e5a13d10d714aaf4b2c59d636690a2df528536267ef61a9f2"}
Sep 09 20:11:29.315744 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:29.315717    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98" event={"ID":"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87","Type":"ContainerStarted","Data":"7524435f33248dfb3c9c7485e23653332d0a696c2366f8ef6aecaba13f4fbb95"}
Sep 09 20:11:29.326432 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:29.326392    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ch7rb" podStartSLOduration=4.103053836 podStartE2EDuration="22.326375569s" totalImagesPullingTime="18.223321733s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:29.32625593 +0000 UTC m=+23.702382676" watchObservedRunningTime="2026-09-09 20:11:29.326375569 +0000 UTC m=+23.702502316"
Sep 09 20:11:30.112345 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:30.112311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:30.112504 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:30.112352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:30.112504 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:30.112429    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:30.112586 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:30.112554    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:30.321010 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:30.320969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98" event={"ID":"1f392cbd-f20d-44e0-8f2f-4f0687cc1f87","Type":"ContainerStarted","Data":"840bdcb3a6208e4ed9271209d6c5ee3b401d1661eb188f3b808ab1fff393b343"}
Sep 09 20:11:30.326460 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:30.326434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx" event={"ID":"8f9801ea-4417-4cd9-9c0f-afde235d5fe5","Type":"ContainerStarted","Data":"cbe0034a525b627d94a21bad8d9b127dca790e25256e6064b09b489d2fba19a8"}
Sep 09 20:11:30.386210 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:30.386178    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:30.386800 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:30.386780    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:30.401017 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:30.400978    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5g98" podStartSLOduration=3.131696147 podStartE2EDuration="23.400960249s" totalImagesPullingTime="20.269264102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:30.33678955 +0000 UTC m=+24.712916297" watchObservedRunningTime="2026-09-09 20:11:30.400960249 +0000 UTC m=+24.777086995"
Sep 09 20:11:31.112270 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:31.112238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:31.112494 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:31.112350    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:31.329338 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:31.329309    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:31.329950 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:31.329780    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4lspj"
Sep 09 20:11:32.111788 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:32.111758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:32.111968 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:32.111877    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:32.112250 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:32.112224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:32.112405 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:32.112327    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:33.112638 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:33.112456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:33.113346 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:33.112725    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:33.337205 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:33.337166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx" event={"ID":"8f9801ea-4417-4cd9-9c0f-afde235d5fe5","Type":"ContainerStarted","Data":"0bbf46e6c6913f6b002582333592559ffcdc7c35b6ced7c15e71ca54747419c0"}
Sep 09 20:11:33.337410 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:33.337381    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:33.337481 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:33.337453    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:33.339768 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:33.339742    2579 generic.go:356] "Generic (PLEG): container finished" podID="a49fb02e-529d-46c8-9f79-3f023c94f54a" containerID="3db7b229d1be7f3ecdb06806d62778be3a5716805637590fc33cb2fa3e057a20" exitCode=0
Sep 09 20:11:33.339882 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:33.339832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd985" event={"ID":"a49fb02e-529d-46c8-9f79-3f023c94f54a","Type":"ContainerDied","Data":"3db7b229d1be7f3ecdb06806d62778be3a5716805637590fc33cb2fa3e057a20"}
Sep 09 20:11:33.353588 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:33.353569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:33.353715 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:33.353701    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:33.363476 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:33.363438    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx" podStartSLOduration=8.080194947 podStartE2EDuration="26.363427664s" totalImagesPullingTime="18.283232717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:33.362318763 +0000 UTC m=+27.738445507" watchObservedRunningTime="2026-09-09 20:11:33.363427664 +0000 UTC m=+27.739554408"
Sep 09 20:11:34.112177 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:34.112112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:34.112366 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:34.112274    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:34.112366 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:34.112327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:34.112481 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:34.112456    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:34.344327 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:34.344254    2579 generic.go:356] "Generic (PLEG): container finished" podID="a49fb02e-529d-46c8-9f79-3f023c94f54a" containerID="d59ac0dcec9939b5f440174281dc5057d1696e80df8220f2c4d00c0c6270165e" exitCode=0
Sep 09 20:11:34.344754 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:34.344333    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd985" event={"ID":"a49fb02e-529d-46c8-9f79-3f023c94f54a","Type":"ContainerDied","Data":"d59ac0dcec9939b5f440174281dc5057d1696e80df8220f2c4d00c0c6270165e"}
Sep 09 20:11:34.344906 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:34.344886    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:11:34.906695 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:34.906458    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5l4sk"]
Sep 09 20:11:34.906881 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:34.906810    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:34.906950 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:34.906916    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:34.907190 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:34.907150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7clz9"]
Sep 09 20:11:34.907244 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:34.907223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:34.907321 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:34.907304    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:34.908793 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:34.908760    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v2fjq"]
Sep 09 20:11:34.908909 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:34.908883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:34.909011 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:34.908990    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:35.234387 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:35.234355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:35.323469 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:35.323444    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:35.348727 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:35.348702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd985" event={"ID":"a49fb02e-529d-46c8-9f79-3f023c94f54a","Type":"ContainerStarted","Data":"ef50db00b08576b67efc4f26c16e17f7c1c169016d0e89743afe38ca78b94f45"}
Sep 09 20:11:35.424548 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:35.424518    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:36.112589 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:36.112559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:36.112745 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:36.112630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:36.112745 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:36.112654    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:36.112745 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:36.112715    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:36.112839 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:36.112758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:36.112839 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:36.112818    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:36.354039 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:36.354004    2579 generic.go:356] "Generic (PLEG): container finished" podID="a49fb02e-529d-46c8-9f79-3f023c94f54a" containerID="ef50db00b08576b67efc4f26c16e17f7c1c169016d0e89743afe38ca78b94f45" exitCode=0
Sep 09 20:11:36.354388 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:36.354066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd985" event={"ID":"a49fb02e-529d-46c8-9f79-3f023c94f54a","Type":"ContainerDied","Data":"ef50db00b08576b67efc4f26c16e17f7c1c169016d0e89743afe38ca78b94f45"}
Sep 09 20:11:38.111659 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:38.111624    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:38.111659 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:38.111653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:38.112177 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:38.111624    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:38.112177 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:38.111747    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:38.112177 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:38.111833    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:38.112177 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:38.111921    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:38.673863 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:38.673829    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:38.772279 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:38.772247    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:39.446569 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:39.446537    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:39.499239 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:39.499206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:39.908824 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:39.908788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:39.909006 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:39.908874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:39.909006 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:39.908948    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:39.909006 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:39.908986    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:39.909115 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:39.909015    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret podName:a213458e-bfea-4d69-aa91-e3a039bfa810 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:11.908997485 +0000 UTC m=+66.285124212 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret") pod "global-pull-secret-syncer-5l4sk" (UID: "a213458e-bfea-4d69-aa91-e3a039bfa810") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:11:39.909115 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:39.909039    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs podName:cd1b99c8-4157-4934-b9e3-1b7f51835443 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:11.909021457 +0000 UTC m=+66.285148188 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs") pod "network-metrics-daemon-v2fjq" (UID: "cd1b99c8-4157-4934-b9e3-1b7f51835443") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:11:40.009329 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.009298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcs7h\" (UniqueName: \"kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h\") pod \"network-check-target-7clz9\" (UID: \"16520da4-678a-4a74-bb1e-6931c60e0f75\") " pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:40.009522 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.009503    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:11:40.009569 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.009531    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:11:40.009569 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.009545    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lcs7h for pod openshift-network-diagnostics/network-check-target-7clz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:40.009642 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.009600    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h podName:16520da4-678a-4a74-bb1e-6931c60e0f75 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:12.009584226 +0000 UTC m=+66.385710953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lcs7h" (UniqueName: "kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h") pod "network-check-target-7clz9" (UID: "16520da4-678a-4a74-bb1e-6931c60e0f75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:11:40.111627 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.111582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:40.111627 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.111621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:40.111835 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.111590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:40.111835 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.111716    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5l4sk" podUID="a213458e-bfea-4d69-aa91-e3a039bfa810"
Sep 09 20:11:40.111835 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.111819    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v2fjq" podUID="cd1b99c8-4157-4934-b9e3-1b7f51835443"
Sep 09 20:11:40.111978 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.111907    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7clz9" podUID="16520da4-678a-4a74-bb1e-6931c60e0f75"
Sep 09 20:11:40.454221 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.454197    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-109.ec2.internal" event="NodeReady"
Sep 09 20:11:40.454518 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.454307    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:11:40.489397 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.489371    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78b59dd4c7-4xtr2"]
Sep 09 20:11:40.503887 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.503863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.504002 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.503885    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k7j2s"]
Sep 09 20:11:40.506031 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.506001    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4lhjh\""
Sep 09 20:11:40.506390 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.506151    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:11:40.506482 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.506255    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:11:40.506482 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.506360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:11:40.512620 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.512364    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:11:40.518268 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.518248    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ccvjh"]
Sep 09 20:11:40.518370 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.518357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.520361 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.520323    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:11:40.520361 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.520334    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7zh2q\""
Sep 09 20:11:40.520488 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.520328    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:11:40.532920 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.532903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78b59dd4c7-4xtr2"]
Sep 09 20:11:40.533018 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.532936    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k7j2s"]
Sep 09 20:11:40.533018 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.532967    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ccvjh"]
Sep 09 20:11:40.533098 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.533029    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ccvjh"
Sep 09 20:11:40.535102 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.535084    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:11:40.535343 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.535325    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xhs4j\""
Sep 09 20:11:40.535423 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.535406    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:11:40.535547 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.535527    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:11:40.614222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-image-registry-private-configuration\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.614355 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8325bdd1-d65b-4dde-933f-83a3950f220c-ca-trust-extracted\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.614355 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.614355 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-installation-pull-secrets\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.614355 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-bound-sa-token\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.614473 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-trusted-ca\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.614473 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.614473 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq2n6\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-kube-api-access-bq2n6\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.614558 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dt2l\" (UniqueName: \"kubernetes.io/projected/f161b80c-3b9a-4256-82d1-b3dae8a7a5d6-kube-api-access-7dt2l\") pod \"ingress-canary-ccvjh\" (UID: \"f161b80c-3b9a-4256-82d1-b3dae8a7a5d6\") " pod="openshift-ingress-canary/ingress-canary-ccvjh"
Sep 09 20:11:40.614558 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58cb6577-3053-40db-a53e-fff4d4a4a172-config-volume\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.614558 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58cb6577-3053-40db-a53e-fff4d4a4a172-tmp-dir\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.614558 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-certificates\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.614558 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f161b80c-3b9a-4256-82d1-b3dae8a7a5d6-cert\") pod \"ingress-canary-ccvjh\" (UID: \"f161b80c-3b9a-4256-82d1-b3dae8a7a5d6\") " pod="openshift-ingress-canary/ingress-canary-ccvjh"
Sep 09 20:11:40.614694 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.614567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sknx7\" (UniqueName: \"kubernetes.io/projected/58cb6577-3053-40db-a53e-fff4d4a4a172-kube-api-access-sknx7\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.714857 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.714800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sknx7\" (UniqueName: \"kubernetes.io/projected/58cb6577-3053-40db-a53e-fff4d4a4a172-kube-api-access-sknx7\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.714857 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.714842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-image-registry-private-configuration\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.715015 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.714862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8325bdd1-d65b-4dde-933f-83a3950f220c-ca-trust-extracted\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.715015 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.714898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.715015 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.714926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-installation-pull-secrets\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.715015 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.714947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-bound-sa-token\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.715015 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.714973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-trusted-ca\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.715015 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.715001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.715324 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.715028    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:40.715324 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.715085    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:40.715324 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.715098    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b59dd4c7-4xtr2: secret "image-registry-tls" not found
Sep 09 20:11:40.715324 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.715098    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls podName:58cb6577-3053-40db-a53e-fff4d4a4a172 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:41.21507782 +0000 UTC m=+35.591204562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls") pod "dns-default-k7j2s" (UID: "58cb6577-3053-40db-a53e-fff4d4a4a172") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:40.715324 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:40.715199    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls podName:8325bdd1-d65b-4dde-933f-83a3950f220c nodeName:}" failed. No retries permitted until 2026-09-09 20:11:41.215178293 +0000 UTC m=+35.591305018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls") pod "image-registry-78b59dd4c7-4xtr2" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c") : secret "image-registry-tls" not found
Sep 09 20:11:40.715324 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.715237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8325bdd1-d65b-4dde-933f-83a3950f220c-ca-trust-extracted\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.715324 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.715243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq2n6\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-kube-api-access-bq2n6\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.715324 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.715301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dt2l\" (UniqueName: \"kubernetes.io/projected/f161b80c-3b9a-4256-82d1-b3dae8a7a5d6-kube-api-access-7dt2l\") pod \"ingress-canary-ccvjh\" (UID: \"f161b80c-3b9a-4256-82d1-b3dae8a7a5d6\") " pod="openshift-ingress-canary/ingress-canary-ccvjh"
Sep 09 20:11:40.715687 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.715343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58cb6577-3053-40db-a53e-fff4d4a4a172-config-volume\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.715687 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.715367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58cb6577-3053-40db-a53e-fff4d4a4a172-tmp-dir\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.715687 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.715402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-certificates\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.715687 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.715430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f161b80c-3b9a-4256-82d1-b3dae8a7a5d6-cert\") pod \"ingress-canary-ccvjh\" (UID: \"f161b80c-3b9a-4256-82d1-b3dae8a7a5d6\") " pod="openshift-ingress-canary/ingress-canary-ccvjh"
Sep 09 20:11:40.715856 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.715776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58cb6577-3053-40db-a53e-fff4d4a4a172-tmp-dir\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.715949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.715928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-trusted-ca\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.716036 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.716016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-certificates\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.716075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.716049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58cb6577-3053-40db-a53e-fff4d4a4a172-config-volume\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.719422 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.719403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-image-registry-private-configuration\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.719528 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.719483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f161b80c-3b9a-4256-82d1-b3dae8a7a5d6-cert\") pod \"ingress-canary-ccvjh\" (UID: \"f161b80c-3b9a-4256-82d1-b3dae8a7a5d6\") " pod="openshift-ingress-canary/ingress-canary-ccvjh"
Sep 09 20:11:40.719528 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.719518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-installation-pull-secrets\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.726106 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.726086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sknx7\" (UniqueName: \"kubernetes.io/projected/58cb6577-3053-40db-a53e-fff4d4a4a172-kube-api-access-sknx7\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:40.726239 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.726204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-bound-sa-token\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.726369 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.726351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq2n6\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-kube-api-access-bq2n6\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:40.726459 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.726441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dt2l\" (UniqueName: \"kubernetes.io/projected/f161b80c-3b9a-4256-82d1-b3dae8a7a5d6-kube-api-access-7dt2l\") pod \"ingress-canary-ccvjh\" (UID: \"f161b80c-3b9a-4256-82d1-b3dae8a7a5d6\") " pod="openshift-ingress-canary/ingress-canary-ccvjh"
Sep 09 20:11:40.840756 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:40.840732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ccvjh"
Sep 09 20:11:41.170795 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:41.170763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ccvjh"]
Sep 09 20:11:41.175829 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:41.175801    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf161b80c_3b9a_4256_82d1_b3dae8a7a5d6.slice/crio-e2361020c7e17990585df7c4505f72134a96de3d05218f22a56f69086d86bb42 WatchSource:0}: Error finding container e2361020c7e17990585df7c4505f72134a96de3d05218f22a56f69086d86bb42: Status 404 returned error can't find the container with id e2361020c7e17990585df7c4505f72134a96de3d05218f22a56f69086d86bb42
Sep 09 20:11:41.220064 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:41.220007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:41.220064 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:41.220051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:41.220217 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:41.220160    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:41.220217 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:41.220164    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:41.220217 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:41.220171    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b59dd4c7-4xtr2: secret "image-registry-tls" not found
Sep 09 20:11:41.220318 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:41.220222    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls podName:58cb6577-3053-40db-a53e-fff4d4a4a172 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.220208313 +0000 UTC m=+36.596335036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls") pod "dns-default-k7j2s" (UID: "58cb6577-3053-40db-a53e-fff4d4a4a172") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:41.220318 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:41.220239    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls podName:8325bdd1-d65b-4dde-933f-83a3950f220c nodeName:}" failed. No retries permitted until 2026-09-09 20:11:42.2202321 +0000 UTC m=+36.596358823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls") pod "image-registry-78b59dd4c7-4xtr2" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c") : secret "image-registry-tls" not found
Sep 09 20:11:41.365567 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:41.365530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ccvjh" event={"ID":"f161b80c-3b9a-4256-82d1-b3dae8a7a5d6","Type":"ContainerStarted","Data":"e2361020c7e17990585df7c4505f72134a96de3d05218f22a56f69086d86bb42"}
Sep 09 20:11:41.368319 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:41.368283    2579 generic.go:356] "Generic (PLEG): container finished" podID="a49fb02e-529d-46c8-9f79-3f023c94f54a" containerID="927f06e107fc417553069d7a11764c185c4fd3fe98289b27f218e9207439710d" exitCode=0
Sep 09 20:11:41.368503 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:41.368326    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd985" event={"ID":"a49fb02e-529d-46c8-9f79-3f023c94f54a","Type":"ContainerDied","Data":"927f06e107fc417553069d7a11764c185c4fd3fe98289b27f218e9207439710d"}
Sep 09 20:11:42.111878 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.111841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:11:42.112755 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.111893    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:11:42.112755 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.111965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:11:42.115680 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.115460    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:11:42.115809 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.115473    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:11:42.115809 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.115473    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:11:42.115809 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.115489    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-52lgp\""
Sep 09 20:11:42.115978 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.115501    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dttj\""
Sep 09 20:11:42.115978 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.115536    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:11:42.228554 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.228513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:42.228736 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.228614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:42.228736 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:42.228717    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:42.228736 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:42.228721    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:42.228736 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:42.228738    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b59dd4c7-4xtr2: secret "image-registry-tls" not found
Sep 09 20:11:42.228955 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:42.228783    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls podName:58cb6577-3053-40db-a53e-fff4d4a4a172 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:44.228765699 +0000 UTC m=+38.604892427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls") pod "dns-default-k7j2s" (UID: "58cb6577-3053-40db-a53e-fff4d4a4a172") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:42.228955 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:42.228797    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls podName:8325bdd1-d65b-4dde-933f-83a3950f220c nodeName:}" failed. No retries permitted until 2026-09-09 20:11:44.228791287 +0000 UTC m=+38.604918013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls") pod "image-registry-78b59dd4c7-4xtr2" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c") : secret "image-registry-tls" not found
Sep 09 20:11:42.374273 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.374235    2579 generic.go:356] "Generic (PLEG): container finished" podID="a49fb02e-529d-46c8-9f79-3f023c94f54a" containerID="4dd3e8f4d99222ffd0f807869d26a2964f78a42cd24bebdef701bbddb22e73b0" exitCode=0
Sep 09 20:11:42.374434 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.374303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd985" event={"ID":"a49fb02e-529d-46c8-9f79-3f023c94f54a","Type":"ContainerDied","Data":"4dd3e8f4d99222ffd0f807869d26a2964f78a42cd24bebdef701bbddb22e73b0"}
Sep 09 20:11:42.461243 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.461216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:42.514000 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:42.513975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:43.379749 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:43.379715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd985" event={"ID":"a49fb02e-529d-46c8-9f79-3f023c94f54a","Type":"ContainerStarted","Data":"6981d24c023fa6af9ebaadffa036308d1ec81fb63244979f4a3365f7e7f5282a"}
Sep 09 20:11:43.400221 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:43.400178    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fd985" podStartSLOduration=7.725714253 podStartE2EDuration="36.400163998s" totalImagesPullingTime="28.674449745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:43.399205822 +0000 UTC m=+37.775332567" watchObservedRunningTime="2026-09-09 20:11:43.400163998 +0000 UTC m=+37.776290743"
Sep 09 20:11:43.519223 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:43.519197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:43.621509 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:43.621479    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:44.245909 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.245873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:44.246057 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.245918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:44.246057 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:44.246034    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:44.246057 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:44.246044    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:44.246057 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:44.246056    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b59dd4c7-4xtr2: secret "image-registry-tls" not found
Sep 09 20:11:44.246201 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:44.246106    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls podName:8325bdd1-d65b-4dde-933f-83a3950f220c nodeName:}" failed. No retries permitted until 2026-09-09 20:11:48.246088887 +0000 UTC m=+42.622215626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls") pod "image-registry-78b59dd4c7-4xtr2" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c") : secret "image-registry-tls" not found
Sep 09 20:11:44.246201 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:44.246118    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls podName:58cb6577-3053-40db-a53e-fff4d4a4a172 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:48.246112915 +0000 UTC m=+42.622239638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls") pod "dns-default-k7j2s" (UID: "58cb6577-3053-40db-a53e-fff4d4a4a172") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:44.383492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.383463    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ccvjh" event={"ID":"f161b80c-3b9a-4256-82d1-b3dae8a7a5d6","Type":"ContainerStarted","Data":"1c9c833c9cfa079ed99c3937aab7ef69165fbb7af418d0c8f7d51533dd49ed12"}
Sep 09 20:11:44.886487 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.886439    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ccvjh" podStartSLOduration=2.5591192 podStartE2EDuration="4.886421953s" totalImagesPullingTime="2.327302753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:44.398850259 +0000 UTC m=+38.774977016" watchObservedRunningTime="2026-09-09 20:11:44.886421953 +0000 UTC m=+39.262548725"
Sep 09 20:11:44.886652 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.886633    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sq74v"]
Sep 09 20:11:44.903775 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.903748    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:44.904192 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.904174    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sq74v"]
Sep 09 20:11:44.904249 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.904238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:44.906477 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.906457    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:11:44.906608 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.906457    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:11:44.906608 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.906457    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:11:44.906608 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.906518    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7dnt6\""
Sep 09 20:11:44.906608 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.906577    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:11:44.958830 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:44.958807    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:45.052648 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.052614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.052754 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.052674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.052754 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.052710    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kz2p\" (UniqueName: \"kubernetes.io/projected/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-api-access-8kz2p\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.052754 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.052730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4a8601a-75a0-4ff6-9937-f757ca1bc286-crio-socket\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.052864 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.052766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4a8601a-75a0-4ff6-9937-f757ca1bc286-data-volume\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.153340 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.153269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4a8601a-75a0-4ff6-9937-f757ca1bc286-data-volume\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.153340 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.153328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.153527 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.153385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.153527 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.153409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kz2p\" (UniqueName: \"kubernetes.io/projected/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-api-access-8kz2p\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.153527 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.153439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4a8601a-75a0-4ff6-9937-f757ca1bc286-crio-socket\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.153527 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:45.153498    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:11:45.153527 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:45.153509    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:45.153683 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:45.153577    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls podName:e4a8601a-75a0-4ff6-9937-f757ca1bc286 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:45.653559847 +0000 UTC m=+40.029686590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq74v" (UID: "e4a8601a-75a0-4ff6-9937-f757ca1bc286") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:45.153683 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:45.153592    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm podName:e4a8601a-75a0-4ff6-9937-f757ca1bc286 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:45.653585914 +0000 UTC m=+40.029712637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq74v" (UID: "e4a8601a-75a0-4ff6-9937-f757ca1bc286") : configmap "kube-rbac-proxy" not found
Sep 09 20:11:45.153683 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.153619    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4a8601a-75a0-4ff6-9937-f757ca1bc286-data-volume\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.153683 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.153667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4a8601a-75a0-4ff6-9937-f757ca1bc286-crio-socket\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.163222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.163199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kz2p\" (UniqueName: \"kubernetes.io/projected/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-api-access-8kz2p\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.178261 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.178235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:45.575487 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.575273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:45.629435 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.629406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:45.656926 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.656895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.657053 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.656948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:45.657053 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:45.657016    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:11:45.657201 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:45.657059    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:45.657201 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:45.657063    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm podName:e4a8601a-75a0-4ff6-9937-f757ca1bc286 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:46.657050869 +0000 UTC m=+41.033177594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq74v" (UID: "e4a8601a-75a0-4ff6-9937-f757ca1bc286") : configmap "kube-rbac-proxy" not found
Sep 09 20:11:45.657201 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:45.657165    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls podName:e4a8601a-75a0-4ff6-9937-f757ca1bc286 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:46.65712151 +0000 UTC m=+41.033248240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq74v" (UID: "e4a8601a-75a0-4ff6-9937-f757ca1bc286") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:45.715258 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.715229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:45.808161 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.808113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:45.905436 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:45.905416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:46.395378 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:46.395344    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:46.452806 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:46.452774    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:46.582463 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:46.582435    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:46.664992 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:46.664932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:46.664992 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:46.664978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:46.665173 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:46.665082    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:11:46.665173 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:46.665096    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:46.665233 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:46.665180    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm podName:e4a8601a-75a0-4ff6-9937-f757ca1bc286 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:48.665160974 +0000 UTC m=+43.041287701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq74v" (UID: "e4a8601a-75a0-4ff6-9937-f757ca1bc286") : configmap "kube-rbac-proxy" not found
Sep 09 20:11:46.665233 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:46.665194    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls podName:e4a8601a-75a0-4ff6-9937-f757ca1bc286 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:48.665188805 +0000 UTC m=+43.041315529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq74v" (UID: "e4a8601a-75a0-4ff6-9937-f757ca1bc286") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:46.683397 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:46.683380    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:46.875856 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:46.875836    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f5dz5_6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16/dns-node-resolver/0.log"
Sep 09 20:11:47.472334 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:47.472295    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:47.523396 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:47.523364    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:47.676909 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:47.676885    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8mqdk_580818db-cd82-48dc-8ed9-4b7c631e9606/node-ca/0.log"
Sep 09 20:11:48.276808 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:48.276766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:48.276993 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:48.276831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:48.276993 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:48.276929    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:11:48.276993 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:48.276983    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:11:48.277100 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:48.276995    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls podName:58cb6577-3053-40db-a53e-fff4d4a4a172 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:56.276980488 +0000 UTC m=+50.653107216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls") pod "dns-default-k7j2s" (UID: "58cb6577-3053-40db-a53e-fff4d4a4a172") : secret "dns-default-metrics-tls" not found
Sep 09 20:11:48.277100 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:48.276998    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78b59dd4c7-4xtr2: secret "image-registry-tls" not found
Sep 09 20:11:48.277100 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:48.277041    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls podName:8325bdd1-d65b-4dde-933f-83a3950f220c nodeName:}" failed. No retries permitted until 2026-09-09 20:11:56.277027227 +0000 UTC m=+50.653153952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls") pod "image-registry-78b59dd4c7-4xtr2" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c") : secret "image-registry-tls" not found
Sep 09 20:11:48.677519 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:48.677499    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ccvjh_f161b80c-3b9a-4256-82d1-b3dae8a7a5d6/serve-healthcheck-canary/0.log"
Sep 09 20:11:48.679638 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:48.679618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:48.679683 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:48.679670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:48.679748 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:48.679732    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:11:48.679748 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:48.679739    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:48.679855 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:48.679790    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls podName:e4a8601a-75a0-4ff6-9937-f757ca1bc286 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:52.679775729 +0000 UTC m=+47.055902452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq74v" (UID: "e4a8601a-75a0-4ff6-9937-f757ca1bc286") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:48.679855 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:48.679805    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm podName:e4a8601a-75a0-4ff6-9937-f757ca1bc286 nodeName:}" failed. No retries permitted until 2026-09-09 20:11:52.679798346 +0000 UTC m=+47.055925069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq74v" (UID: "e4a8601a-75a0-4ff6-9937-f757ca1bc286") : configmap "kube-rbac-proxy" not found
Sep 09 20:11:48.697483 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:48.697460    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:49.202820 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:49.202786    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:51.205974 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:51.205940    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:51.863140 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:51.863105    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:52.008020 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:52.007992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:52.155161 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:52.155113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:52.345576 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:52.345541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:52.452002 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:52.451926    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:52.585418 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:52.585389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:52.691749 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:52.691725    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:52.708904 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:52.708856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:52.708977 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:52.708945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:52.709091 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:52.709077    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:52.709164 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:11:52.709153    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls podName:e4a8601a-75a0-4ff6-9937-f757ca1bc286 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:00.709140393 +0000 UTC m=+55.085267130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq74v" (UID: "e4a8601a-75a0-4ff6-9937-f757ca1bc286") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:11:52.709467 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:52.709450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4a8601a-75a0-4ff6-9937-f757ca1bc286-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:11:52.786603 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:52.786576    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:53.749037 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:53.749006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:54.107150 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:54.107048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:54.315799 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:54.315737    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:54.506626 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:54.506588    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:54.905738 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:54.905712    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:55.279184 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:55.279101    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:55.509935 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:55.509910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:55.704245 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:55.704213    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:55.813370 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:55.813338    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:55.914255 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:55.914221    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:56.309163 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.309116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:56.336279 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.336259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:56.336387 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.336306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:56.338897 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.338873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58cb6577-3053-40db-a53e-fff4d4a4a172-metrics-tls\") pod \"dns-default-k7j2s\" (UID: \"58cb6577-3053-40db-a53e-fff4d4a4a172\") " pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:56.338996 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.338927    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls\") pod \"image-registry-78b59dd4c7-4xtr2\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") " pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:56.415332 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.415307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:56.426703 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.426688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:56.507397 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.507365    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:56.557799 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.557765    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:56.694714 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.694689    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78b59dd4c7-4xtr2"]
Sep 09 20:11:56.697814 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:56.697788    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8325bdd1_d65b_4dde_933f_83a3950f220c.slice/crio-c9377a4f02b32f7a83bb72bd61994f49b5ce0ccfab7afd1659a449331ae39d84 WatchSource:0}: Error finding container c9377a4f02b32f7a83bb72bd61994f49b5ce0ccfab7afd1659a449331ae39d84: Status 404 returned error can't find the container with id c9377a4f02b32f7a83bb72bd61994f49b5ce0ccfab7afd1659a449331ae39d84
Sep 09 20:11:56.714438 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.714360    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k7j2s"]
Sep 09 20:11:56.718362 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:11:56.718341    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58cb6577_3053_40db_a53e_fff4d4a4a172.slice/crio-df80e9b7c86eb89969e10da9071d97ce3895d1a29ce1010c10ca7a9bffdc7d02 WatchSource:0}: Error finding container df80e9b7c86eb89969e10da9071d97ce3895d1a29ce1010c10ca7a9bffdc7d02: Status 404 returned error can't find the container with id df80e9b7c86eb89969e10da9071d97ce3895d1a29ce1010c10ca7a9bffdc7d02
Sep 09 20:11:56.905251 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:56.905217    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:57.415505 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:57.415457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k7j2s" event={"ID":"58cb6577-3053-40db-a53e-fff4d4a4a172","Type":"ContainerStarted","Data":"df80e9b7c86eb89969e10da9071d97ce3895d1a29ce1010c10ca7a9bffdc7d02"}
Sep 09 20:11:57.417040 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:57.417002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2" event={"ID":"8325bdd1-d65b-4dde-933f-83a3950f220c","Type":"ContainerStarted","Data":"a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff"}
Sep 09 20:11:57.417194 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:57.417055    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2" event={"ID":"8325bdd1-d65b-4dde-933f-83a3950f220c","Type":"ContainerStarted","Data":"c9377a4f02b32f7a83bb72bd61994f49b5ce0ccfab7afd1659a449331ae39d84"}
Sep 09 20:11:57.417194 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:57.417171    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:11:57.437631 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:57.437590    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2" podStartSLOduration=28.437577761 podStartE2EDuration="28.437577761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:57.435987447 +0000 UTC m=+51.812114196" watchObservedRunningTime="2026-09-09 20:11:57.437577761 +0000 UTC m=+51.813704506"
Sep 09 20:11:57.512264 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:57.512227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:57.911185 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:57.911157    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:58.103999 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:58.103961    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:58.315071 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:58.314989    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:58.509231 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:58.509203    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:58.919276 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:58.919238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:59.424601 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:59.424567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k7j2s" event={"ID":"58cb6577-3053-40db-a53e-fff4d4a4a172","Type":"ContainerStarted","Data":"bf33af723863e11b747b8124ac603a6118b73ec5f97991f752ee49db3e540f7e"}
Sep 09 20:11:59.424779 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:59.424630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k7j2s" event={"ID":"58cb6577-3053-40db-a53e-fff4d4a4a172","Type":"ContainerStarted","Data":"3e24128b9184825b8fe4352ced6e21591a82b8f4996e94b7d50e2ae6cb39b731"}
Sep 09 20:11:59.424779 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:59.424720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:11:59.440615 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:59.440572    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k7j2s" podStartSLOduration=17.795343745 podStartE2EDuration="19.440559735s" totalImagesPullingTime="1.64521599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:11:59.439315526 +0000 UTC m=+53.815442270" watchObservedRunningTime="2026-09-09 20:11:59.440559735 +0000 UTC m=+53.816686481"
Sep 09 20:11:59.710390 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:59.710304    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:11:59.907684 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:11:59.907661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:00.307986 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:00.307956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:00.507742 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:00.507710    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:00.763388 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:00.763347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:12:00.765693 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:00.765675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4a8601a-75a0-4ff6-9937-f757ca1bc286-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq74v\" (UID: \"e4a8601a-75a0-4ff6-9937-f757ca1bc286\") " pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:12:00.813253 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:00.813232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sq74v"
Sep 09 20:12:00.906525 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:00.906494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:01.078560 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:01.078477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sq74v"]
Sep 09 20:12:01.081213 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:12:01.081179    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a8601a_75a0_4ff6_9937_f757ca1bc286.slice/crio-50b72dd4f2bc82f92813b00fe70ff8a2e362cc48f73e221b734c511cac90c90c WatchSource:0}: Error finding container 50b72dd4f2bc82f92813b00fe70ff8a2e362cc48f73e221b734c511cac90c90c: Status 404 returned error can't find the container with id 50b72dd4f2bc82f92813b00fe70ff8a2e362cc48f73e221b734c511cac90c90c
Sep 09 20:12:01.431170 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:01.431119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq74v" event={"ID":"e4a8601a-75a0-4ff6-9937-f757ca1bc286","Type":"ContainerStarted","Data":"651d963111a9422a21d049c9942f4ce812479bca69e332315a87fbcbf61fa35e"}
Sep 09 20:12:01.431370 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:01.431201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq74v" event={"ID":"e4a8601a-75a0-4ff6-9937-f757ca1bc286","Type":"ContainerStarted","Data":"50b72dd4f2bc82f92813b00fe70ff8a2e362cc48f73e221b734c511cac90c90c"}
Sep 09 20:12:01.509217 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:01.509181    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:01.610475 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:01.610447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:01.665519 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:01.665492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:01.721030 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:01.720940    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:02.436059 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:02.436028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq74v" event={"ID":"e4a8601a-75a0-4ff6-9937-f757ca1bc286","Type":"ContainerStarted","Data":"6f0953bc0f681c3fd24206dad3656a95b73a36f821c1a295bcf27cd408558b69"}
Sep 09 20:12:02.603606 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:02.603567    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:02.701991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:02.701889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:02.759183 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:02.759148    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:04.445100 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:04.445066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq74v" event={"ID":"e4a8601a-75a0-4ff6-9937-f757ca1bc286","Type":"ContainerStarted","Data":"01081463be1eac483e65a72737fc92d2bc9361cebd2b532ba46af72eb6d4b645"}
Sep 09 20:12:04.461991 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:04.461945    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sq74v" podStartSLOduration=18.014425195 podStartE2EDuration="20.46190818s" totalImagesPullingTime="2.447482985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:04.461506035 +0000 UTC m=+58.837632778" watchObservedRunningTime="2026-09-09 20:12:04.46190818 +0000 UTC m=+58.838034924"
Sep 09 20:12:05.359565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:05.359533    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rk4kx"
Sep 09 20:12:08.826566 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.826531    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"]
Sep 09 20:12:08.829172 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.829155    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"]
Sep 09 20:12:08.829269 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.829252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:08.830934 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.830915    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:08.831570 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.831553    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 20:12:08.832400 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.832363    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:12:08.832492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.832423    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:12:08.832896 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.832877    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:12:08.833181 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.833164    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 20:12:08.833296 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.833277    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 20:12:08.833352 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.833300    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 20:12:08.833352 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.833321    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 20:12:08.842615 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.842595    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78b59dd4c7-4xtr2"]
Sep 09 20:12:08.843354 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.843330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"]
Sep 09 20:12:08.847400 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.847373    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"]
Sep 09 20:12:08.913324 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.913301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-ca\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:08.913445 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.913331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3660e36e-5618-4151-ac15-34198cf8ba15-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:08.913445 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.913363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee4693b4-0eb3-4f5a-8084-da3e69291079-klusterlet-config\") pod \"klusterlet-addon-workmgr-8f47f8f6d-rkcrz\" (UID: \"ee4693b4-0eb3-4f5a-8084-da3e69291079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:08.913445 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.913381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdk82\" (UniqueName: \"kubernetes.io/projected/3660e36e-5618-4151-ac15-34198cf8ba15-kube-api-access-wdk82\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:08.913445 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.913440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:08.913661 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.913474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:08.913661 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.913567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdll\" (UniqueName: \"kubernetes.io/projected/ee4693b4-0eb3-4f5a-8084-da3e69291079-kube-api-access-lpdll\") pod \"klusterlet-addon-workmgr-8f47f8f6d-rkcrz\" (UID: \"ee4693b4-0eb3-4f5a-8084-da3e69291079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:08.913661 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.913600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-hub\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:08.913661 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.913635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee4693b4-0eb3-4f5a-8084-da3e69291079-tmp\") pod \"klusterlet-addon-workmgr-8f47f8f6d-rkcrz\" (UID: \"ee4693b4-0eb3-4f5a-8084-da3e69291079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:08.942331 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:08.942310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:09.014322 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.014293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.014322 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.014321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.014534 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.014353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdll\" (UniqueName: \"kubernetes.io/projected/ee4693b4-0eb3-4f5a-8084-da3e69291079-kube-api-access-lpdll\") pod \"klusterlet-addon-workmgr-8f47f8f6d-rkcrz\" (UID: \"ee4693b4-0eb3-4f5a-8084-da3e69291079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:09.014534 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.014367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-hub\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.014534 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.014387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee4693b4-0eb3-4f5a-8084-da3e69291079-tmp\") pod \"klusterlet-addon-workmgr-8f47f8f6d-rkcrz\" (UID: \"ee4693b4-0eb3-4f5a-8084-da3e69291079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:09.014668 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.014575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-ca\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.014668 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.014617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3660e36e-5618-4151-ac15-34198cf8ba15-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.014847 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.014667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee4693b4-0eb3-4f5a-8084-da3e69291079-klusterlet-config\") pod \"klusterlet-addon-workmgr-8f47f8f6d-rkcrz\" (UID: \"ee4693b4-0eb3-4f5a-8084-da3e69291079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:09.014847 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.014696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdk82\" (UniqueName: \"kubernetes.io/projected/3660e36e-5618-4151-ac15-34198cf8ba15-kube-api-access-wdk82\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.014847 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.014749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ee4693b4-0eb3-4f5a-8084-da3e69291079-tmp\") pod \"klusterlet-addon-workmgr-8f47f8f6d-rkcrz\" (UID: \"ee4693b4-0eb3-4f5a-8084-da3e69291079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:09.015372 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.015348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3660e36e-5618-4151-ac15-34198cf8ba15-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.017098 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.017074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.017566 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.017540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.017642 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.017625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-ca\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.017701 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.017681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3660e36e-5618-4151-ac15-34198cf8ba15-hub\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.017776 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.017759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ee4693b4-0eb3-4f5a-8084-da3e69291079-klusterlet-config\") pod \"klusterlet-addon-workmgr-8f47f8f6d-rkcrz\" (UID: \"ee4693b4-0eb3-4f5a-8084-da3e69291079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:09.027550 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.027528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdll\" (UniqueName: \"kubernetes.io/projected/ee4693b4-0eb3-4f5a-8084-da3e69291079-kube-api-access-lpdll\") pod \"klusterlet-addon-workmgr-8f47f8f6d-rkcrz\" (UID: \"ee4693b4-0eb3-4f5a-8084-da3e69291079\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:09.031721 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.031699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdk82\" (UniqueName: \"kubernetes.io/projected/3660e36e-5618-4151-ac15-34198cf8ba15-kube-api-access-wdk82\") pod \"cluster-proxy-proxy-agent-7b7589998d-2pzp8\" (UID: \"3660e36e-5618-4151-ac15-34198cf8ba15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.054376 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.054345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:09.139037 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.139016    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:09.152297 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.152275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"
Sep 09 20:12:09.154330 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.154309    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:09.253453 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.253419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:09.404116 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.404039    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"]
Sep 09 20:12:09.407224 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:12:09.407191    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4693b4_0eb3_4f5a_8084_da3e69291079.slice/crio-7c481246f838e532a86beb28ace027bedda52d704a90c0943dff9a163ffb7342 WatchSource:0}: Error finding container 7c481246f838e532a86beb28ace027bedda52d704a90c0943dff9a163ffb7342: Status 404 returned error can't find the container with id 7c481246f838e532a86beb28ace027bedda52d704a90c0943dff9a163ffb7342
Sep 09 20:12:09.426275 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.426253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8"]
Sep 09 20:12:09.428782 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:12:09.428747    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3660e36e_5618_4151_ac15_34198cf8ba15.slice/crio-574a3ddc07d79a30b246117fbdaee1f84c31ba75e6a1bfdf2f456ecb7e216aa7 WatchSource:0}: Error finding container 574a3ddc07d79a30b246117fbdaee1f84c31ba75e6a1bfdf2f456ecb7e216aa7: Status 404 returned error can't find the container with id 574a3ddc07d79a30b246117fbdaee1f84c31ba75e6a1bfdf2f456ecb7e216aa7
Sep 09 20:12:09.428962 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.428946    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k7j2s"
Sep 09 20:12:09.461482 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.461436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8" event={"ID":"3660e36e-5618-4151-ac15-34198cf8ba15","Type":"ContainerStarted","Data":"574a3ddc07d79a30b246117fbdaee1f84c31ba75e6a1bfdf2f456ecb7e216aa7"}
Sep 09 20:12:09.462621 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:09.462600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz" event={"ID":"ee4693b4-0eb3-4f5a-8084-da3e69291079","Type":"ContainerStarted","Data":"7c481246f838e532a86beb28ace027bedda52d704a90c0943dff9a163ffb7342"}
Sep 09 20:12:11.159901 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:11.159840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:11.429848 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:11.429731    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:11.948489 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:11.948285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:12:11.948868 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:11.948760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:12:11.951188 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:11.951165    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:12:11.951964 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:11.951790    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:12:11.961899 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:11.961853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd1b99c8-4157-4934-b9e3-1b7f51835443-metrics-certs\") pod \"network-metrics-daemon-v2fjq\" (UID: \"cd1b99c8-4157-4934-b9e3-1b7f51835443\") " pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:12:11.962555 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:11.962533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a213458e-bfea-4d69-aa91-e3a039bfa810-original-pull-secret\") pod \"global-pull-secret-syncer-5l4sk\" (UID: \"a213458e-bfea-4d69-aa91-e3a039bfa810\") " pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:12:12.050004 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:12.049966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcs7h\" (UniqueName: \"kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h\") pod \"network-check-target-7clz9\" (UID: \"16520da4-678a-4a74-bb1e-6931c60e0f75\") " pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:12:12.053206 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:12.053182    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:12:12.063244 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:12.063033    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:12:12.079458 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:12.079424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcs7h\" (UniqueName: \"kubernetes.io/projected/16520da4-678a-4a74-bb1e-6931c60e0f75-kube-api-access-lcs7h\") pod \"network-check-target-7clz9\" (UID: \"16520da4-678a-4a74-bb1e-6931c60e0f75\") " pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:12:12.123433 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:12.123402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5l4sk"
Sep 09 20:12:12.133832 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:12.133637    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-52lgp\""
Sep 09 20:12:12.137223 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:12.137024    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9dttj\""
Sep 09 20:12:12.141256 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:12.141170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:12:12.145247 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:12.145169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v2fjq"
Sep 09 20:12:13.744161 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:13.744099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:13.848265 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:13.848237    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:13.931897 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:13.931865    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:14.111973 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:14.111898    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:14.417366 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:14.417332    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:15.152561 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.152490    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7clz9"]
Sep 09 20:12:15.155419 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.155358    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5l4sk"]
Sep 09 20:12:15.155932 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:12:15.155910    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16520da4_678a_4a74_bb1e_6931c60e0f75.slice/crio-9a168e023e9b8a144fdbdb03d4b181c2d36c1ca4a54eea4f64b1f4441b7791b0 WatchSource:0}: Error finding container 9a168e023e9b8a144fdbdb03d4b181c2d36c1ca4a54eea4f64b1f4441b7791b0: Status 404 returned error can't find the container with id 9a168e023e9b8a144fdbdb03d4b181c2d36c1ca4a54eea4f64b1f4441b7791b0
Sep 09 20:12:15.156324 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.156305    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v2fjq"]
Sep 09 20:12:15.159154 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:12:15.159109    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda213458e_bfea_4d69_aa91_e3a039bfa810.slice/crio-0fd5a5ea06c174cdba8fb48d00262dba1d6eda5b1a9f1c59db88133f44896f40 WatchSource:0}: Error finding container 0fd5a5ea06c174cdba8fb48d00262dba1d6eda5b1a9f1c59db88133f44896f40: Status 404 returned error can't find the container with id 0fd5a5ea06c174cdba8fb48d00262dba1d6eda5b1a9f1c59db88133f44896f40
Sep 09 20:12:15.159706 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:12:15.159685    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1b99c8_4157_4934_b9e3_1b7f51835443.slice/crio-017a6fb93d610f067a6f0c37a9e2fae6acf5f3e976b54d0ee7ba0e0288a3d599 WatchSource:0}: Error finding container 017a6fb93d610f067a6f0c37a9e2fae6acf5f3e976b54d0ee7ba0e0288a3d599: Status 404 returned error can't find the container with id 017a6fb93d610f067a6f0c37a9e2fae6acf5f3e976b54d0ee7ba0e0288a3d599
Sep 09 20:12:15.487973 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.487889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5l4sk" event={"ID":"a213458e-bfea-4d69-aa91-e3a039bfa810","Type":"ContainerStarted","Data":"0fd5a5ea06c174cdba8fb48d00262dba1d6eda5b1a9f1c59db88133f44896f40"}
Sep 09 20:12:15.489186 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.489159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7clz9" event={"ID":"16520da4-678a-4a74-bb1e-6931c60e0f75","Type":"ContainerStarted","Data":"9a168e023e9b8a144fdbdb03d4b181c2d36c1ca4a54eea4f64b1f4441b7791b0"}
Sep 09 20:12:15.491347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.491316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8" event={"ID":"3660e36e-5618-4151-ac15-34198cf8ba15","Type":"ContainerStarted","Data":"e471b105dbf764121545cba49798301348ef85fc35071cc554a7ceacc13daac0"}
Sep 09 20:12:15.491449 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.491354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8" event={"ID":"3660e36e-5618-4151-ac15-34198cf8ba15","Type":"ContainerStarted","Data":"6931bee4d8355d69c9bb9779b8ca267caf9cefdaa03b76ae227bf7f1e9521bf0"}
Sep 09 20:12:15.491449 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.491372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8" event={"ID":"3660e36e-5618-4151-ac15-34198cf8ba15","Type":"ContainerStarted","Data":"12622b0224e48947bfb110606852685938b47e0bdbe039e04f19aebcef7d1280"}
Sep 09 20:12:15.492694 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.492671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz" event={"ID":"ee4693b4-0eb3-4f5a-8084-da3e69291079","Type":"ContainerStarted","Data":"eab842031779290309935c39c7a19bd5ecbe39cbcffb0b06da221a9bf6698708"}
Sep 09 20:12:15.492801 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.492784    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:15.493954 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.493933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2fjq" event={"ID":"cd1b99c8-4157-4934-b9e3-1b7f51835443","Type":"ContainerStarted","Data":"017a6fb93d610f067a6f0c37a9e2fae6acf5f3e976b54d0ee7ba0e0288a3d599"}
Sep 09 20:12:15.494370 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.494354    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz"
Sep 09 20:12:15.512456 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.512416    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b7589998d-2pzp8" podStartSLOduration=2.050855003 podStartE2EDuration="7.512387293s" totalImagesPullingTime="5.46153229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:15.510906696 +0000 UTC m=+69.887033434" watchObservedRunningTime="2026-09-09 20:12:15.512387293 +0000 UTC m=+69.888514046"
Sep 09 20:12:15.527090 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:15.527033    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8f47f8f6d-rkcrz" podStartSLOduration=2.040389302 podStartE2EDuration="7.527017603s" totalImagesPullingTime="5.486628301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:15.526182795 +0000 UTC m=+69.902309540" watchObservedRunningTime="2026-09-09 20:12:15.527017603 +0000 UTC m=+69.903144350"
Sep 09 20:12:16.500177 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:16.500077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2fjq" event={"ID":"cd1b99c8-4157-4934-b9e3-1b7f51835443","Type":"ContainerStarted","Data":"6fbd6a48b1266a9213d12514e97ac16d9eb2f9ebe665f68c1074d8db78d6db43"}
Sep 09 20:12:16.500177 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:16.500157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v2fjq" event={"ID":"cd1b99c8-4157-4934-b9e3-1b7f51835443","Type":"ContainerStarted","Data":"f22927d74e36dfb749fa27b18ab25bc2334a7bee6e2720b1cab0debe5955a533"}
Sep 09 20:12:16.517160 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:16.517088    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v2fjq" podStartSLOduration=68.453061903 podStartE2EDuration="1m9.517072132s" totalImagesPullingTime="1.064010229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:16.515992669 +0000 UTC m=+70.892119419" watchObservedRunningTime="2026-09-09 20:12:16.517072132 +0000 UTC m=+70.893198896"
Sep 09 20:12:16.588695 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:16.588635    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:16.688948 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:16.688910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:16.743246 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:16.743215    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:16.849161 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:16.849062    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:16.950680 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:16.950651    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:17.485355 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.485316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:17.540333 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.540299    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:17.932512 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.932469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:17.963328 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.963297    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wssxb"]
Sep 09 20:12:17.965503 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.965485    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:17.967930 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.967909    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 20:12:17.969641 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.968968    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:12:17.969641 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.969004    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:12:17.969641 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.969028    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:12:17.969641 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.969005    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jcc6s\""
Sep 09 20:12:17.969641 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.969075    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 20:12:17.973572 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.973555    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wssxb"]
Sep 09 20:12:17.996809 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:17.996784    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:18.080390 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.080349    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:18.100467 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.100432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/68683f39-29de-42b4-b847-5e87eec34ca0-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.100605 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.100496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68683f39-29de-42b4-b847-5e87eec34ca0-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.100651 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.100594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68683f39-29de-42b4-b847-5e87eec34ca0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.100651 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.100635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvwhs\" (UniqueName: \"kubernetes.io/projected/68683f39-29de-42b4-b847-5e87eec34ca0-kube-api-access-hvwhs\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.201155 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.201054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68683f39-29de-42b4-b847-5e87eec34ca0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.201155 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.201121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvwhs\" (UniqueName: \"kubernetes.io/projected/68683f39-29de-42b4-b847-5e87eec34ca0-kube-api-access-hvwhs\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.201336 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.201197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/68683f39-29de-42b4-b847-5e87eec34ca0-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.201336 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.201238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68683f39-29de-42b4-b847-5e87eec34ca0-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.201961 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.201938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68683f39-29de-42b4-b847-5e87eec34ca0-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.204026 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.204000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68683f39-29de-42b4-b847-5e87eec34ca0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.204151 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.204110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/68683f39-29de-42b4-b847-5e87eec34ca0-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.209751 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.209723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvwhs\" (UniqueName: \"kubernetes.io/projected/68683f39-29de-42b4-b847-5e87eec34ca0-kube-api-access-hvwhs\") pod \"prometheus-operator-6f4946db56-wssxb\" (UID: \"68683f39-29de-42b4-b847-5e87eec34ca0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.276605 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.276567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb"
Sep 09 20:12:18.551542 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.551449    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:18.651331 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.651293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:18.722349 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.722303    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:18.807601 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.807482    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:18.848851 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:18.848822    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:12:19.030485 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:19.030423    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:19.190904 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:19.190858    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:19.386883 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:19.386838    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:19.487088 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:19.487006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:19.811675 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:19.811646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:19.913513 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:19.913478    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:19.945726 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:19.945702    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-wssxb"]
Sep 09 20:12:19.948569 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:12:19.948544    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68683f39_29de_42b4_b847_5e87eec34ca0.slice/crio-5987154488ac5c08b8885a1a1a7a1b31f1736ea12f86a26833e33672635d494c WatchSource:0}: Error finding container 5987154488ac5c08b8885a1a1a7a1b31f1736ea12f86a26833e33672635d494c: Status 404 returned error can't find the container with id 5987154488ac5c08b8885a1a1a7a1b31f1736ea12f86a26833e33672635d494c
Sep 09 20:12:20.019992 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:20.019925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:20.115258 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:20.115232    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:20.516281 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:20.516250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb" event={"ID":"68683f39-29de-42b4-b847-5e87eec34ca0","Type":"ContainerStarted","Data":"5987154488ac5c08b8885a1a1a7a1b31f1736ea12f86a26833e33672635d494c"}
Sep 09 20:12:20.518050 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:20.518019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5l4sk" event={"ID":"a213458e-bfea-4d69-aa91-e3a039bfa810","Type":"ContainerStarted","Data":"944f05c0a785e2e80b670d1c9469a6977fb46c9c95ac21e1a66ff152155049dd"}
Sep 09 20:12:20.519525 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:20.519501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7clz9" event={"ID":"16520da4-678a-4a74-bb1e-6931c60e0f75","Type":"ContainerStarted","Data":"fae40f45e80dad10fbb11ee6719cb0e9375d569d28eed9d2660306dbcdb29464"}
Sep 09 20:12:20.519684 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:20.519666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:12:20.535062 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:20.535019    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5l4sk" podStartSLOduration=69.00338285 podStartE2EDuration="1m13.535002506s" totalImagesPullingTime="4.531619656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:20.534224162 +0000 UTC m=+74.910350905" watchObservedRunningTime="2026-09-09 20:12:20.535002506 +0000 UTC m=+74.911129253"
Sep 09 20:12:20.550414 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:20.550372    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7clz9" podStartSLOduration=69.023794822 podStartE2EDuration="1m13.55035459s" totalImagesPullingTime="4.526559768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:20.549560665 +0000 UTC m=+74.925687416" watchObservedRunningTime="2026-09-09 20:12:20.55035459 +0000 UTC m=+74.926481339"
Sep 09 20:12:20.619507 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:20.619471    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:20.803293 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:20.803221    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:21.250138 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:21.250107    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:21.488198 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:21.488088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:21.525869 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:21.525819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb" event={"ID":"68683f39-29de-42b4-b847-5e87eec34ca0","Type":"ContainerStarted","Data":"70a26d4e7395749a56d8d4e1639a523d2dc2537207ee89fcb674f97b7abfdb27"}
Sep 09 20:12:21.526002 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:21.525896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb" event={"ID":"68683f39-29de-42b4-b847-5e87eec34ca0","Type":"ContainerStarted","Data":"b2630ebfcd7387c0ca5428e05d9d5236975031d98df78324524fc5870593a060"}
Sep 09 20:12:21.544965 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:21.544918    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-wssxb" podStartSLOduration=3.262755993 podStartE2EDuration="4.5449031s" totalImagesPullingTime="1.282147107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:21.543609612 +0000 UTC m=+75.919736356" watchObservedRunningTime="2026-09-09 20:12:21.5449031 +0000 UTC m=+75.921029846"
Sep 09 20:12:22.219277 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:22.219240    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:22.334709 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:22.334676    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:23.074664 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.074629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:23.310435 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.310397    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:23.365456 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.365432    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f5b5q"]
Sep 09 20:12:23.368584 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.368564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.370518 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.370499    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:12:23.370659 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.370641    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zl8h2\""
Sep 09 20:12:23.370904 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.370877    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:12:23.370904 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.370899    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:12:23.436172 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.436149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/386ad5c3-303f-4a5d-9111-34f6dc9865d5-metrics-client-ca\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.436267 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.436179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzmz7\" (UniqueName: \"kubernetes.io/projected/386ad5c3-303f-4a5d-9111-34f6dc9865d5-kube-api-access-pzmz7\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.436267 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.436204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-wtmp\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.436267 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.436230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.436399 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.436276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-tls\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.436399 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.436294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/386ad5c3-303f-4a5d-9111-34f6dc9865d5-sys\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.436399 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.436331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.436399 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.436359    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/386ad5c3-303f-4a5d-9111-34f6dc9865d5-root\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.436399 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.436377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-textfile\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.438303 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.438285    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:23.536695 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.536669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-wtmp\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.536826 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.536700    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.536826 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.536729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-tls\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.536826 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.536806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/386ad5c3-303f-4a5d-9111-34f6dc9865d5-sys\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.536984 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:12:23.536833    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:12:23.536984 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.536841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.536984 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.536859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-wtmp\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.536984 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.536873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/386ad5c3-303f-4a5d-9111-34f6dc9865d5-root\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.536984 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.536896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-textfile\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.536984 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:12:23.536932    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-tls podName:386ad5c3-303f-4a5d-9111-34f6dc9865d5 nodeName:}" failed. No retries permitted until 2026-09-09 20:12:24.036899355 +0000 UTC m=+78.413026089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-tls") pod "node-exporter-f5b5q" (UID: "386ad5c3-303f-4a5d-9111-34f6dc9865d5") : secret "node-exporter-tls" not found
Sep 09 20:12:23.536984 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.536937    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/386ad5c3-303f-4a5d-9111-34f6dc9865d5-root\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.537284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.537025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/386ad5c3-303f-4a5d-9111-34f6dc9865d5-metrics-client-ca\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.537284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.537058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzmz7\" (UniqueName: \"kubernetes.io/projected/386ad5c3-303f-4a5d-9111-34f6dc9865d5-kube-api-access-pzmz7\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.537284 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.537143    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:23.537384 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.537306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-textfile\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.537435 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.537419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/386ad5c3-303f-4a5d-9111-34f6dc9865d5-sys\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.537556 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.537520    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.537591 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.537551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/386ad5c3-303f-4a5d-9111-34f6dc9865d5-metrics-client-ca\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.539149 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.539114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.548091 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.548044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzmz7\" (UniqueName: \"kubernetes.io/projected/386ad5c3-303f-4a5d-9111-34f6dc9865d5-kube-api-access-pzmz7\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:23.638544 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.638480    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:23.737171 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.737118    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:23.839518 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.839493    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:23.941739 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:23.941680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:24.038538 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:24.038507    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:24.040337 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:24.040312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-tls\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:24.042662 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:24.042641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/386ad5c3-303f-4a5d-9111-34f6dc9865d5-node-exporter-tls\") pod \"node-exporter-f5b5q\" (UID: \"386ad5c3-303f-4a5d-9111-34f6dc9865d5\") " pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:24.277333 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:24.277256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f5b5q"
Sep 09 20:12:24.288486 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:12:24.288457    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386ad5c3_303f_4a5d_9111_34f6dc9865d5.slice/crio-5e91739c5165de3323a562334a3cd62a6b1bccc249e33681bde4f446294dd6b3 WatchSource:0}: Error finding container 5e91739c5165de3323a562334a3cd62a6b1bccc249e33681bde4f446294dd6b3: Status 404 returned error can't find the container with id 5e91739c5165de3323a562334a3cd62a6b1bccc249e33681bde4f446294dd6b3
Sep 09 20:12:24.298877 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:24.298850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:24.381187 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:24.381156    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:24.537896 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:24.537817    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5b5q" event={"ID":"386ad5c3-303f-4a5d-9111-34f6dc9865d5","Type":"ContainerStarted","Data":"5e91739c5165de3323a562334a3cd62a6b1bccc249e33681bde4f446294dd6b3"}
Sep 09 20:12:24.851810 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:24.851738    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:24.952850 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:24.952818    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:25.545626 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:25.545543    2579 generic.go:356] "Generic (PLEG): container finished" podID="386ad5c3-303f-4a5d-9111-34f6dc9865d5" containerID="74091cd39fbab424fd781742f1bf5c99e1ffd5b12221dc077ab533118c5f0f81" exitCode=0
Sep 09 20:12:25.545626 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:25.545591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5b5q" event={"ID":"386ad5c3-303f-4a5d-9111-34f6dc9865d5","Type":"ContainerDied","Data":"74091cd39fbab424fd781742f1bf5c99e1ffd5b12221dc077ab533118c5f0f81"}
Sep 09 20:12:25.924229 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:25.924193    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:26.134706 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:26.134675    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:26.264670 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:26.264590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:26.367391 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:26.367354    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:26.551411 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:26.551324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5b5q" event={"ID":"386ad5c3-303f-4a5d-9111-34f6dc9865d5","Type":"ContainerStarted","Data":"7e644aa918dcdebce9594f857e85afde4fa577a95ff1916a945e6bc3ea2f68e8"}
Sep 09 20:12:26.551411 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:26.551378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f5b5q" event={"ID":"386ad5c3-303f-4a5d-9111-34f6dc9865d5","Type":"ContainerStarted","Data":"acae4b20b8997b3df524cf233a1c657cc626e401ebe81da48e755d44985a46f2"}
Sep 09 20:12:26.573857 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:26.573813    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f5b5q" podStartSLOduration=2.538550837 podStartE2EDuration="3.573800085s" totalImagesPullingTime="967.348817ms" totalInitContainerRuntime="67.900431ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:26.573661368 +0000 UTC m=+80.949788131" watchObservedRunningTime="2026-09-09 20:12:26.573800085 +0000 UTC m=+80.949926829"
Sep 09 20:12:26.625818 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:26.625787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:26.682873 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:26.682842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:28.084332 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:28.084235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:28.154307 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:28.154274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:28.473671 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:28.473644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:29.520987 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:29.520952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:29.885140 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:29.885088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:29.940085 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:29.940037    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:30.824475 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:30.824416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:30.875683 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:30.875649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:31.479829 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:31.479798    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:31.599097 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:31.599065    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:31.652305 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:31.652283    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:31.847244 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:31.847175    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:31.947932 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:31.947905    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:33.863967 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:33.863930    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2" podUID="8325bdd1-d65b-4dde-933f-83a3950f220c" containerName="registry" containerID="cri-o://a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff" gracePeriod=30
Sep 09 20:12:34.100192 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.100167    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:12:34.218358 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.218272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq2n6\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-kube-api-access-bq2n6\") pod \"8325bdd1-d65b-4dde-933f-83a3950f220c\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") "
Sep 09 20:12:34.218358 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.218316    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-image-registry-private-configuration\") pod \"8325bdd1-d65b-4dde-933f-83a3950f220c\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") "
Sep 09 20:12:34.218358 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.218338    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-certificates\") pod \"8325bdd1-d65b-4dde-933f-83a3950f220c\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") "
Sep 09 20:12:34.218358 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.218362    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-installation-pull-secrets\") pod \"8325bdd1-d65b-4dde-933f-83a3950f220c\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") "
Sep 09 20:12:34.218657 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.218396    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls\") pod \"8325bdd1-d65b-4dde-933f-83a3950f220c\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") "
Sep 09 20:12:34.218657 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.218430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8325bdd1-d65b-4dde-933f-83a3950f220c-ca-trust-extracted\") pod \"8325bdd1-d65b-4dde-933f-83a3950f220c\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") "
Sep 09 20:12:34.218657 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.218452    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-trusted-ca\") pod \"8325bdd1-d65b-4dde-933f-83a3950f220c\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") "
Sep 09 20:12:34.218657 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.218482    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-bound-sa-token\") pod \"8325bdd1-d65b-4dde-933f-83a3950f220c\" (UID: \"8325bdd1-d65b-4dde-933f-83a3950f220c\") "
Sep 09 20:12:34.219530 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.219202    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8325bdd1-d65b-4dde-933f-83a3950f220c" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:34.219530 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.219470    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8325bdd1-d65b-4dde-933f-83a3950f220c" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:12:34.221168 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.221100    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-kube-api-access-bq2n6" (OuterVolumeSpecName: "kube-api-access-bq2n6") pod "8325bdd1-d65b-4dde-933f-83a3950f220c" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c"). InnerVolumeSpecName "kube-api-access-bq2n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:34.221269 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.221182    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8325bdd1-d65b-4dde-933f-83a3950f220c" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:34.221323 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.221301    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8325bdd1-d65b-4dde-933f-83a3950f220c" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:12:34.221603 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.221542    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8325bdd1-d65b-4dde-933f-83a3950f220c" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:34.221603 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.221592    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8325bdd1-d65b-4dde-933f-83a3950f220c" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:12:34.227423 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.227395    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8325bdd1-d65b-4dde-933f-83a3950f220c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8325bdd1-d65b-4dde-933f-83a3950f220c" (UID: "8325bdd1-d65b-4dde-933f-83a3950f220c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:12:34.320904 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.320865    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.320904 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.320900    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8325bdd1-d65b-4dde-933f-83a3950f220c-ca-trust-extracted\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.320904 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.320910    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-trusted-ca\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.321097 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.320918    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-bound-sa-token\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.321097 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.320927    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq2n6\" (UniqueName: \"kubernetes.io/projected/8325bdd1-d65b-4dde-933f-83a3950f220c-kube-api-access-bq2n6\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.321097 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.320940    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-image-registry-private-configuration\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.321097 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.320949    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8325bdd1-d65b-4dde-933f-83a3950f220c-registry-certificates\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.321097 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.320956    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8325bdd1-d65b-4dde-933f-83a3950f220c-installation-pull-secrets\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:12:34.584262 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.584171    2579 generic.go:356] "Generic (PLEG): container finished" podID="8325bdd1-d65b-4dde-933f-83a3950f220c" containerID="a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff" exitCode=0
Sep 09 20:12:34.584262 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.584212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2" event={"ID":"8325bdd1-d65b-4dde-933f-83a3950f220c","Type":"ContainerDied","Data":"a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff"}
Sep 09 20:12:34.584262 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.584238    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2"
Sep 09 20:12:34.584655 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.584273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78b59dd4c7-4xtr2" event={"ID":"8325bdd1-d65b-4dde-933f-83a3950f220c","Type":"ContainerDied","Data":"c9377a4f02b32f7a83bb72bd61994f49b5ce0ccfab7afd1659a449331ae39d84"}
Sep 09 20:12:34.584655 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.584293    2579 scope.go:117] "RemoveContainer" containerID="a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff"
Sep 09 20:12:34.595244 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.595228    2579 scope.go:117] "RemoveContainer" containerID="a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff"
Sep 09 20:12:34.595575 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:12:34.595544    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff\": container with ID starting with a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff not found: ID does not exist" containerID="a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff"
Sep 09 20:12:34.595648 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.595583    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff"} err="failed to get container status \"a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff\": rpc error: code = NotFound desc = could not find container \"a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff\": container with ID starting with a9db3a09710592c1c891d10c5ab491644536e9e97fe34cb7e56926f2abe30cff not found: ID does not exist"
Sep 09 20:12:34.613653 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.613622    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78b59dd4c7-4xtr2"]
Sep 09 20:12:34.618298 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:34.618278    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78b59dd4c7-4xtr2"]
Sep 09 20:12:36.024697 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:36.024659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:36.078757 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:36.078723    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:36.117892 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:36.117851    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8325bdd1-d65b-4dde-933f-83a3950f220c" path="/var/lib/kubelet/pods/8325bdd1-d65b-4dde-933f-83a3950f220c/volumes"
Sep 09 20:12:36.566212 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:36.566183    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:38.122894 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:38.122867    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:38.182417 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:38.182387    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:40.210025 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.209992    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9dd54d69b-rhrmb"]
Sep 09 20:12:40.210410 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.210218    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8325bdd1-d65b-4dde-933f-83a3950f220c" containerName="registry"
Sep 09 20:12:40.210410 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.210229    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8325bdd1-d65b-4dde-933f-83a3950f220c" containerName="registry"
Sep 09 20:12:40.210410 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.210280    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8325bdd1-d65b-4dde-933f-83a3950f220c" containerName="registry"
Sep 09 20:12:40.212920 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.212901    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.215209 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.215185    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:12:40.215893 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.215876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jh2nm\""
Sep 09 20:12:40.216004 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.215936    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:12:40.216064 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.216005    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:12:40.216064 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.216026    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:12:40.216064 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.216005    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:12:40.216231 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.216188    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:12:40.216231 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.216195    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:12:40.219847 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.219828    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:12:40.223507 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.223488    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9dd54d69b-rhrmb"]
Sep 09 20:12:40.362209 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.362166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-oauth-serving-cert\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.362209 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.362211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-trusted-ca-bundle\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.362463 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.362228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkptg\" (UniqueName: \"kubernetes.io/projected/dffcadd5-8280-4e44-89d6-f95cf11c4809-kube-api-access-lkptg\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.362463 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.362283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-serving-cert\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.362463 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.362319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-service-ca\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.362463 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.362355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-oauth-config\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.362463 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.362383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-config\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.463540 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.463458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-oauth-config\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.463540 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.463500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-config\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.463734 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.463551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-oauth-serving-cert\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.463734 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.463575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-trusted-ca-bundle\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.463734 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.463595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkptg\" (UniqueName: \"kubernetes.io/projected/dffcadd5-8280-4e44-89d6-f95cf11c4809-kube-api-access-lkptg\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.463734 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.463693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-serving-cert\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.463734 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.463716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-service-ca\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.464418 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.464397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-config\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.464542 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.464486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-oauth-serving-cert\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.464604 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.464545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-trusted-ca-bundle\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.464700 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.464660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-service-ca\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.466114 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.466093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-oauth-config\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.466463 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.466444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-serving-cert\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.471507 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.471487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkptg\" (UniqueName: \"kubernetes.io/projected/dffcadd5-8280-4e44-89d6-f95cf11c4809-kube-api-access-lkptg\") pod \"console-9dd54d69b-rhrmb\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") " pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.522184 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.522158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:40.791740 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.791649    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9dd54d69b-rhrmb"]
Sep 09 20:12:40.795623 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:12:40.795598    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffcadd5_8280_4e44_89d6_f95cf11c4809.slice/crio-9d03ca280557ca5f562341727145ce6645fee21c8357ed6da925d4fb25adbd59 WatchSource:0}: Error finding container 9d03ca280557ca5f562341727145ce6645fee21c8357ed6da925d4fb25adbd59: Status 404 returned error can't find the container with id 9d03ca280557ca5f562341727145ce6645fee21c8357ed6da925d4fb25adbd59
Sep 09 20:12:40.813873 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.813847    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:40.876868 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:40.876844    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:41.611443 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:41.611404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9dd54d69b-rhrmb" event={"ID":"dffcadd5-8280-4e44-89d6-f95cf11c4809","Type":"ContainerStarted","Data":"9d03ca280557ca5f562341727145ce6645fee21c8357ed6da925d4fb25adbd59"}
Sep 09 20:12:43.620063 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:43.620030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9dd54d69b-rhrmb" event={"ID":"dffcadd5-8280-4e44-89d6-f95cf11c4809","Type":"ContainerStarted","Data":"a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930"}
Sep 09 20:12:43.637360 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:43.637317    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9dd54d69b-rhrmb" podStartSLOduration=1.074987719 podStartE2EDuration="3.637301687s" totalImagesPullingTime="2.562313968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:43.635970383 +0000 UTC m=+98.012097128" watchObservedRunningTime="2026-09-09 20:12:43.637301687 +0000 UTC m=+98.013428455"
Sep 09 20:12:46.594809 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:46.594762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:46.692467 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:46.692436    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:12:50.522516 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:50.522480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:50.522933 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:50.522561    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:50.527351 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:50.527328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:50.645235 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:50.645211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:12:51.528145 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:51.528103    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7clz9"
Sep 09 20:12:57.914032 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:12:57.913995    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:01.584148 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:01.584096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:01.646366 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:01.646338    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:16.587033 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:16.587002    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:16.688463 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:16.688428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:39.426800 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:39.426762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:39.483165 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:39.483116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:43.969047 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:43.969011    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:44.022662 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:44.022631    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:51.244507 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:51.244473    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:51.298951 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:51.298918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:55.231806 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:55.231776    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9dd54d69b-rhrmb"]
Sep 09 20:13:55.339762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:55.339718    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:55.391588 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:55.391560    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:13:55.443675 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:13:55.443644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:01.589410 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:01.589374    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:01.690065 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:01.690030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:20.255625 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.255582    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9dd54d69b-rhrmb" podUID="dffcadd5-8280-4e44-89d6-f95cf11c4809" containerName="console" containerID="cri-o://a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930" gracePeriod=15
Sep 09 20:14:20.492444 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.492419    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:14:20.564247 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.564160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-oauth-serving-cert\") pod \"dffcadd5-8280-4e44-89d6-f95cf11c4809\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") "
Sep 09 20:14:20.564247 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.564203    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-trusted-ca-bundle\") pod \"dffcadd5-8280-4e44-89d6-f95cf11c4809\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") "
Sep 09 20:14:20.564247 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.564235    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkptg\" (UniqueName: \"kubernetes.io/projected/dffcadd5-8280-4e44-89d6-f95cf11c4809-kube-api-access-lkptg\") pod \"dffcadd5-8280-4e44-89d6-f95cf11c4809\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") "
Sep 09 20:14:20.564526 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.564284    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-oauth-config\") pod \"dffcadd5-8280-4e44-89d6-f95cf11c4809\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") "
Sep 09 20:14:20.564526 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.564307    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-config\") pod \"dffcadd5-8280-4e44-89d6-f95cf11c4809\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") "
Sep 09 20:14:20.564526 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.564326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-serving-cert\") pod \"dffcadd5-8280-4e44-89d6-f95cf11c4809\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") "
Sep 09 20:14:20.564526 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.564345    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-service-ca\") pod \"dffcadd5-8280-4e44-89d6-f95cf11c4809\" (UID: \"dffcadd5-8280-4e44-89d6-f95cf11c4809\") "
Sep 09 20:14:20.564721 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.564656    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dffcadd5-8280-4e44-89d6-f95cf11c4809" (UID: "dffcadd5-8280-4e44-89d6-f95cf11c4809"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:20.564721 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.564715    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dffcadd5-8280-4e44-89d6-f95cf11c4809" (UID: "dffcadd5-8280-4e44-89d6-f95cf11c4809"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:20.564836 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.564814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-service-ca" (OuterVolumeSpecName: "service-ca") pod "dffcadd5-8280-4e44-89d6-f95cf11c4809" (UID: "dffcadd5-8280-4e44-89d6-f95cf11c4809"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:20.564883 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.564809    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-config" (OuterVolumeSpecName: "console-config") pod "dffcadd5-8280-4e44-89d6-f95cf11c4809" (UID: "dffcadd5-8280-4e44-89d6-f95cf11c4809"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:14:20.566780 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.566748    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dffcadd5-8280-4e44-89d6-f95cf11c4809" (UID: "dffcadd5-8280-4e44-89d6-f95cf11c4809"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:20.566904 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.566807    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dffcadd5-8280-4e44-89d6-f95cf11c4809" (UID: "dffcadd5-8280-4e44-89d6-f95cf11c4809"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:14:20.566904 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.566814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffcadd5-8280-4e44-89d6-f95cf11c4809-kube-api-access-lkptg" (OuterVolumeSpecName: "kube-api-access-lkptg") pod "dffcadd5-8280-4e44-89d6-f95cf11c4809" (UID: "dffcadd5-8280-4e44-89d6-f95cf11c4809"). InnerVolumeSpecName "kube-api-access-lkptg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:14:20.665363 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.665328    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:14:20.665363 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.665357    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-serving-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:14:20.665363 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.665367    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-service-ca\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:14:20.665594 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.665379    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-oauth-serving-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:14:20.665594 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.665386    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dffcadd5-8280-4e44-89d6-f95cf11c4809-trusted-ca-bundle\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:14:20.665594 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.665395    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkptg\" (UniqueName: \"kubernetes.io/projected/dffcadd5-8280-4e44-89d6-f95cf11c4809-kube-api-access-lkptg\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:14:20.665594 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.665402    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dffcadd5-8280-4e44-89d6-f95cf11c4809-console-oauth-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:14:20.942900 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.942864    2579 generic.go:356] "Generic (PLEG): container finished" podID="dffcadd5-8280-4e44-89d6-f95cf11c4809" containerID="a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930" exitCode=0
Sep 09 20:14:20.943096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.942907    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9dd54d69b-rhrmb"
Sep 09 20:14:20.943096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.942920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9dd54d69b-rhrmb" event={"ID":"dffcadd5-8280-4e44-89d6-f95cf11c4809","Type":"ContainerDied","Data":"a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930"}
Sep 09 20:14:20.943096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.942970    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9dd54d69b-rhrmb" event={"ID":"dffcadd5-8280-4e44-89d6-f95cf11c4809","Type":"ContainerDied","Data":"9d03ca280557ca5f562341727145ce6645fee21c8357ed6da925d4fb25adbd59"}
Sep 09 20:14:20.943096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.942996    2579 scope.go:117] "RemoveContainer" containerID="a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930"
Sep 09 20:14:20.951308 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.951289    2579 scope.go:117] "RemoveContainer" containerID="a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930"
Sep 09 20:14:20.951568 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:14:20.951551    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930\": container with ID starting with a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930 not found: ID does not exist" containerID="a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930"
Sep 09 20:14:20.951620 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.951577    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930"} err="failed to get container status \"a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930\": rpc error: code = NotFound desc = could not find container \"a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930\": container with ID starting with a0db923756f678e4fca4e76f26ef8544ebb8a1ef42f972ca1438a3cf16ef1930 not found: ID does not exist"
Sep 09 20:14:20.962395 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.962370    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9dd54d69b-rhrmb"]
Sep 09 20:14:20.967022 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:20.966998    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9dd54d69b-rhrmb"]
Sep 09 20:14:22.116583 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:22.116547    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dffcadd5-8280-4e44-89d6-f95cf11c4809" path="/var/lib/kubelet/pods/dffcadd5-8280-4e44-89d6-f95cf11c4809/volumes"
Sep 09 20:14:41.290850 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:41.290814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:14:46.005811 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:14:46.005772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:15:20.897940 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:15:20.897910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:05.306208 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:05.306175    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:06.051101 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:06.051074    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:16:06.568180 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:06.568149    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:32.444011 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:32.443973    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:32.536678 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:32.536644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:37.775423 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:37.775390    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:38.041785 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.041711    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7lt8m"]
Sep 09 20:16:38.041949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.041938    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dffcadd5-8280-4e44-89d6-f95cf11c4809" containerName="console"
Sep 09 20:16:38.041990 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.041952    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dffcadd5-8280-4e44-89d6-f95cf11c4809" containerName="console"
Sep 09 20:16:38.041990 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.041988    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dffcadd5-8280-4e44-89d6-f95cf11c4809" containerName="console"
Sep 09 20:16:38.044594 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.044579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:38.046757 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.046731    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:16:38.047055 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.047034    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:16:38.047162 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.047073    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-jtx85\""
Sep 09 20:16:38.047412 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.047393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:38.047558 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.047543    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 09 20:16:38.047626 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.047549    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:16:38.047680 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.047581    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:16:38.054848 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.054827    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7lt8m"]
Sep 09 20:16:38.151053 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.151022    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:38.176101 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.176077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxhf\" (UniqueName: \"kubernetes.io/projected/9443fe0a-a619-4885-8549-972373bd3bc9-kube-api-access-vgxhf\") pod \"keda-operator-579c8958f7-7lt8m\" (UID: \"9443fe0a-a619-4885-8549-972373bd3bc9\") " pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:38.176204 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.176123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9443fe0a-a619-4885-8549-972373bd3bc9-certificates\") pod \"keda-operator-579c8958f7-7lt8m\" (UID: \"9443fe0a-a619-4885-8549-972373bd3bc9\") " pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:38.176204 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.176166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9443fe0a-a619-4885-8549-972373bd3bc9-cabundle0\") pod \"keda-operator-579c8958f7-7lt8m\" (UID: \"9443fe0a-a619-4885-8549-972373bd3bc9\") " pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:38.250949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.250918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:38.277214 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.277187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9443fe0a-a619-4885-8549-972373bd3bc9-certificates\") pod \"keda-operator-579c8958f7-7lt8m\" (UID: \"9443fe0a-a619-4885-8549-972373bd3bc9\") " pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:38.277329 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.277220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9443fe0a-a619-4885-8549-972373bd3bc9-cabundle0\") pod \"keda-operator-579c8958f7-7lt8m\" (UID: \"9443fe0a-a619-4885-8549-972373bd3bc9\") " pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:38.277329 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.277285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxhf\" (UniqueName: \"kubernetes.io/projected/9443fe0a-a619-4885-8549-972373bd3bc9-kube-api-access-vgxhf\") pod \"keda-operator-579c8958f7-7lt8m\" (UID: \"9443fe0a-a619-4885-8549-972373bd3bc9\") " pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:38.277441 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:16:38.277332    2579 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:16:38.277441 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:16:38.277348    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:16:38.277441 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:16:38.277358    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-7lt8m: references non-existent secret key: ca.crt
Sep 09 20:16:38.277441 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:16:38.277428    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9443fe0a-a619-4885-8549-972373bd3bc9-certificates podName:9443fe0a-a619-4885-8549-972373bd3bc9 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:38.77741073 +0000 UTC m=+333.153537454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9443fe0a-a619-4885-8549-972373bd3bc9-certificates") pod "keda-operator-579c8958f7-7lt8m" (UID: "9443fe0a-a619-4885-8549-972373bd3bc9") : references non-existent secret key: ca.crt
Sep 09 20:16:38.277880 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.277862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/9443fe0a-a619-4885-8549-972373bd3bc9-cabundle0\") pod \"keda-operator-579c8958f7-7lt8m\" (UID: \"9443fe0a-a619-4885-8549-972373bd3bc9\") " pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:38.288188 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.288165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxhf\" (UniqueName: \"kubernetes.io/projected/9443fe0a-a619-4885-8549-972373bd3bc9-kube-api-access-vgxhf\") pod \"keda-operator-579c8958f7-7lt8m\" (UID: \"9443fe0a-a619-4885-8549-972373bd3bc9\") " pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:38.382331 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.382303    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"]
Sep 09 20:16:38.385074 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.385060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:38.387051 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.387033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 09 20:16:38.393111 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.393089    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"]
Sep 09 20:16:38.478367 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.478344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6qff\" (UniqueName: \"kubernetes.io/projected/5558110f-979e-4cb7-8211-beae0cce6cdb-kube-api-access-g6qff\") pod \"keda-metrics-apiserver-77857b7b98-tk54x\" (UID: \"5558110f-979e-4cb7-8211-beae0cce6cdb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:38.478526 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.478390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5558110f-979e-4cb7-8211-beae0cce6cdb-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-tk54x\" (UID: \"5558110f-979e-4cb7-8211-beae0cce6cdb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:38.478526 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.478412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5558110f-979e-4cb7-8211-beae0cce6cdb-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tk54x\" (UID: \"5558110f-979e-4cb7-8211-beae0cce6cdb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:38.490901 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.490877    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:38.579347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.579326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5558110f-979e-4cb7-8211-beae0cce6cdb-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tk54x\" (UID: \"5558110f-979e-4cb7-8211-beae0cce6cdb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:38.579421 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.579375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6qff\" (UniqueName: \"kubernetes.io/projected/5558110f-979e-4cb7-8211-beae0cce6cdb-kube-api-access-g6qff\") pod \"keda-metrics-apiserver-77857b7b98-tk54x\" (UID: \"5558110f-979e-4cb7-8211-beae0cce6cdb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:38.579421 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.579413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5558110f-979e-4cb7-8211-beae0cce6cdb-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-tk54x\" (UID: \"5558110f-979e-4cb7-8211-beae0cce6cdb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:38.579516 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:16:38.579467    2579 secret.go:281] references non-existent secret key: tls.crt
Sep 09 20:16:38.579516 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:16:38.579483    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 09 20:16:38.579516 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:16:38.579505    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x: references non-existent secret key: tls.crt
Sep 09 20:16:38.579643 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:16:38.579569    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5558110f-979e-4cb7-8211-beae0cce6cdb-certificates podName:5558110f-979e-4cb7-8211-beae0cce6cdb nodeName:}" failed. No retries permitted until 2026-09-09 20:16:39.079554269 +0000 UTC m=+333.455680996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5558110f-979e-4cb7-8211-beae0cce6cdb-certificates") pod "keda-metrics-apiserver-77857b7b98-tk54x" (UID: "5558110f-979e-4cb7-8211-beae0cce6cdb") : references non-existent secret key: tls.crt
Sep 09 20:16:38.579694 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.579683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5558110f-979e-4cb7-8211-beae0cce6cdb-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-tk54x\" (UID: \"5558110f-979e-4cb7-8211-beae0cce6cdb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:38.589584 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.589564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6qff\" (UniqueName: \"kubernetes.io/projected/5558110f-979e-4cb7-8211-beae0cce6cdb-kube-api-access-g6qff\") pod \"keda-metrics-apiserver-77857b7b98-tk54x\" (UID: \"5558110f-979e-4cb7-8211-beae0cce6cdb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:38.780856 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.780776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9443fe0a-a619-4885-8549-972373bd3bc9-certificates\") pod \"keda-operator-579c8958f7-7lt8m\" (UID: \"9443fe0a-a619-4885-8549-972373bd3bc9\") " pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:38.783322 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.783296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9443fe0a-a619-4885-8549-972373bd3bc9-certificates\") pod \"keda-operator-579c8958f7-7lt8m\" (UID: \"9443fe0a-a619-4885-8549-972373bd3bc9\") " pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:38.817557 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.817533    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:38.915767 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.915738    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:38.954898 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:38.954864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:39.090081 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:39.089986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5558110f-979e-4cb7-8211-beae0cce6cdb-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tk54x\" (UID: \"5558110f-979e-4cb7-8211-beae0cce6cdb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:39.092650 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:39.092626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5558110f-979e-4cb7-8211-beae0cce6cdb-certificates\") pod \"keda-metrics-apiserver-77857b7b98-tk54x\" (UID: \"5558110f-979e-4cb7-8211-beae0cce6cdb\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:39.159647 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:39.159616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:39.233824 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:39.233782    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-7lt8m"]
Sep 09 20:16:39.237940 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:16:39.237905    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9443fe0a_a619_4885_8549_972373bd3bc9.slice/crio-fafd933232c2e644dceee9c0e934323a2891f8d7a7092efc241d74ee38c01a48 WatchSource:0}: Error finding container fafd933232c2e644dceee9c0e934323a2891f8d7a7092efc241d74ee38c01a48: Status 404 returned error can't find the container with id fafd933232c2e644dceee9c0e934323a2891f8d7a7092efc241d74ee38c01a48
Sep 09 20:16:39.240094 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:39.239650    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:16:39.269946 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:39.269918    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:39.295389 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:39.295356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:39.376328 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:39.376291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-7lt8m" event={"ID":"9443fe0a-a619-4885-8549-972373bd3bc9","Type":"ContainerStarted","Data":"fafd933232c2e644dceee9c0e934323a2891f8d7a7092efc241d74ee38c01a48"}
Sep 09 20:16:39.501534 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:39.501501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:39.573941 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:39.573913    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"]
Sep 09 20:16:39.576757 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:16:39.576730    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5558110f_979e_4cb7_8211_beae0cce6cdb.slice/crio-4de2f0116c09e9c9383bbd8adba9be21b73cb52f80a7cdd3e553817586223789 WatchSource:0}: Error finding container 4de2f0116c09e9c9383bbd8adba9be21b73cb52f80a7cdd3e553817586223789: Status 404 returned error can't find the container with id 4de2f0116c09e9c9383bbd8adba9be21b73cb52f80a7cdd3e553817586223789
Sep 09 20:16:39.658546 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:39.658513    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:40.381767 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:40.381686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x" event={"ID":"5558110f-979e-4cb7-8211-beae0cce6cdb","Type":"ContainerStarted","Data":"4de2f0116c09e9c9383bbd8adba9be21b73cb52f80a7cdd3e553817586223789"}
Sep 09 20:16:43.395397 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:43.395362    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-7lt8m" event={"ID":"9443fe0a-a619-4885-8549-972373bd3bc9","Type":"ContainerStarted","Data":"c26658d9b9f8c76b23ca797983d8930273b018e523177e20fc4633b70f510477"}
Sep 09 20:16:43.396117 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:43.395486    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:16:43.414828 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:43.414777    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-7lt8m" podStartSLOduration=2.151369443 podStartE2EDuration="5.414758045s" totalImagesPullingTime="3.263388602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:43.413017447 +0000 UTC m=+337.789144192" watchObservedRunningTime="2026-09-09 20:16:43.414758045 +0000 UTC m=+337.790884796"
Sep 09 20:16:45.404640 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:45.404603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x" event={"ID":"5558110f-979e-4cb7-8211-beae0cce6cdb","Type":"ContainerStarted","Data":"d89379ce063b07776f7b0daa71c8e918d68d257f6c23aadfa403573d111d375a"}
Sep 09 20:16:45.405006 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:45.404737    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:16:45.420953 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:45.420907    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x" podStartSLOduration=2.641967828 podStartE2EDuration="7.420893646s" totalImagesPullingTime="4.778925818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:45.419652063 +0000 UTC m=+339.795779016" watchObservedRunningTime="2026-09-09 20:16:45.420893646 +0000 UTC m=+339.797020392"
Sep 09 20:16:56.146791 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:56.146742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:56.202583 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:56.202541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:16:56.413110 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:16:56.413039    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-tk54x"
Sep 09 20:17:02.795846 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:02.795816    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:02.894499 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:02.894467    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:04.401530 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:04.401496    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-7lt8m"
Sep 09 20:17:04.772035 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:04.771945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:04.872986 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:04.872958    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:31.110675 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:31.110640    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:31.165869 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:31.165842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:34.492087 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:34.492056    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:34.542658 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:34.542632    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:34.637096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:34.637067    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:34.737884 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:34.737852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:35.236225 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:35.236193    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:45.319460 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:45.319414    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:45.377323 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:45.377290    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:45.434348 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:45.434320    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:55.667122 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.667080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:55.744559 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.744528    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"]
Sep 09 20:17:55.744972 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.744950    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:55.746489 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.746469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"
Sep 09 20:17:55.749871 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.749399    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:17:55.749871 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.749628    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:17:55.749871 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.749647    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:17:55.750184 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.750159    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-xvdg9\""
Sep 09 20:17:55.758307 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.758286    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"]
Sep 09 20:17:55.802861 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.802831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a258b9d8-420e-4d37-82ca-3ffe8618a940-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-svxcw\" (UID: \"a258b9d8-420e-4d37-82ca-3ffe8618a940\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"
Sep 09 20:17:55.802974 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.802869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmjsn\" (UniqueName: \"kubernetes.io/projected/a258b9d8-420e-4d37-82ca-3ffe8618a940-kube-api-access-hmjsn\") pod \"llmisvc-controller-manager-645f9c8fd6-svxcw\" (UID: \"a258b9d8-420e-4d37-82ca-3ffe8618a940\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"
Sep 09 20:17:55.822298 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.822261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:55.904096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.904060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a258b9d8-420e-4d37-82ca-3ffe8618a940-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-svxcw\" (UID: \"a258b9d8-420e-4d37-82ca-3ffe8618a940\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"
Sep 09 20:17:55.904227 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.904108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmjsn\" (UniqueName: \"kubernetes.io/projected/a258b9d8-420e-4d37-82ca-3ffe8618a940-kube-api-access-hmjsn\") pod \"llmisvc-controller-manager-645f9c8fd6-svxcw\" (UID: \"a258b9d8-420e-4d37-82ca-3ffe8618a940\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"
Sep 09 20:17:55.906599 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.906571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a258b9d8-420e-4d37-82ca-3ffe8618a940-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-svxcw\" (UID: \"a258b9d8-420e-4d37-82ca-3ffe8618a940\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"
Sep 09 20:17:55.912454 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.912433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmjsn\" (UniqueName: \"kubernetes.io/projected/a258b9d8-420e-4d37-82ca-3ffe8618a940-kube-api-access-hmjsn\") pod \"llmisvc-controller-manager-645f9c8fd6-svxcw\" (UID: \"a258b9d8-420e-4d37-82ca-3ffe8618a940\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"
Sep 09 20:17:55.923365 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:55.923310    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:56.062142 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:56.062098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"
Sep 09 20:17:56.331373 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:56.331294    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"]
Sep 09 20:17:56.334113 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:17:56.334086    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda258b9d8_420e_4d37_82ca_3ffe8618a940.slice/crio-2ddba266d57ba2df97d5be4926a92f9603ed002c0792390fccc71d5d5a9e8c22 WatchSource:0}: Error finding container 2ddba266d57ba2df97d5be4926a92f9603ed002c0792390fccc71d5d5a9e8c22: Status 404 returned error can't find the container with id 2ddba266d57ba2df97d5be4926a92f9603ed002c0792390fccc71d5d5a9e8c22
Sep 09 20:17:56.482671 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:56.482639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:56.582763 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:56.582695    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:56.648193 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:56.648160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw" event={"ID":"a258b9d8-420e-4d37-82ca-3ffe8618a940","Type":"ContainerStarted","Data":"2ddba266d57ba2df97d5be4926a92f9603ed002c0792390fccc71d5d5a9e8c22"}
Sep 09 20:17:56.682214 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:56.682190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:17:59.661160 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:59.661118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw" event={"ID":"a258b9d8-420e-4d37-82ca-3ffe8618a940","Type":"ContainerStarted","Data":"92557db966f0ce6cda8a6aa037cee97f116ed7bc00ecb318cb7ec1970871f646"}
Sep 09 20:17:59.661555 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:59.661248    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"
Sep 09 20:17:59.675899 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:17:59.675807    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw" podStartSLOduration=1.844865505 podStartE2EDuration="4.675789509s" totalImagesPullingTime="2.830924004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:59.674973819 +0000 UTC m=+414.051100566" watchObservedRunningTime="2026-09-09 20:17:59.675789509 +0000 UTC m=+414.051916253"
Sep 09 20:18:30.666806 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:18:30.666732    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-svxcw"
Sep 09 20:19:08.925078 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:19:08.925043    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:11.925803 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:19:11.925765    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:12.007220 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:19:12.007185    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:12.095240 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:19:12.095206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:12.470306 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:19:12.470271    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:13.512299 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:19:13.512265    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:19:13.563290 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:19:13.563260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:29.226537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:20:29.226501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:45.301824 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:20:45.301791    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:49.406959 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:20:49.406923    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:49.507347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:20:49.507313    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:51.679693 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:20:51.679665    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:51.843474 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:20:51.843432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:20:51.895201 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:20:51.895171    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:00.867775 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.867746    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"]
Sep 09 20:21:00.869862 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.869842    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:00.872558 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.872515    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 20:21:00.872690 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.872563    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:21:00.872690 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.872568    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f4cxz\""
Sep 09 20:21:00.872690 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.872576    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:21:00.873377 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.873349    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 20:21:00.873470 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.873394    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:21:00.873526 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.873496    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:21:00.883796 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.883777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"]
Sep 09 20:21:00.953697 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.953671    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:00.978330 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.978312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:00.978417 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.978353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:00.978417 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.978379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx5kn\" (UniqueName: \"kubernetes.io/projected/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-api-access-kx5kn\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:00.978417 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.978403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:00.978556 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:00.978478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.058465 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.058436    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:01.079165 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.079118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.079275 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.079181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx5kn\" (UniqueName: \"kubernetes.io/projected/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-api-access-kx5kn\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.079275 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.079204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.079275 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.079227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.079415 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.079275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.079690 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.079667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.079879 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.079864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.079918 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.079899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.081648 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.081630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-proxy-tls\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.087093 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.087069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx5kn\" (UniqueName: \"kubernetes.io/projected/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-api-access-kx5kn\") pod \"isvc-canary-rollback-predictor-6477dbf789-4km2p\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.155745 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.155686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:01.179845 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.179824    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:01.448198 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:01.447962    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"]
Sep 09 20:21:01.450787 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:21:01.450761    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f8e0bc_ed07_40f9_a1a1_33f2b6326a85.slice/crio-76011cb828ddb0175cb521d5fc8854fbc7eb4b2a6598628d5923987939222d50 WatchSource:0}: Error finding container 76011cb828ddb0175cb521d5fc8854fbc7eb4b2a6598628d5923987939222d50: Status 404 returned error can't find the container with id 76011cb828ddb0175cb521d5fc8854fbc7eb4b2a6598628d5923987939222d50
Sep 09 20:21:02.259938 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:02.259890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" event={"ID":"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85","Type":"ContainerStarted","Data":"76011cb828ddb0175cb521d5fc8854fbc7eb4b2a6598628d5923987939222d50"}
Sep 09 20:21:06.275528 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:06.275494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" event={"ID":"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85","Type":"ContainerStarted","Data":"56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a"}
Sep 09 20:21:09.286544 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:09.286513    2579 generic.go:356] "Generic (PLEG): container finished" podID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerID="56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a" exitCode=0
Sep 09 20:21:09.286940 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:09.286569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" event={"ID":"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85","Type":"ContainerDied","Data":"56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a"}
Sep 09 20:21:13.569429 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:13.569381    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:21:23.347120 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:23.347039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" event={"ID":"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85","Type":"ContainerStarted","Data":"d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c"}
Sep 09 20:21:25.358325 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:25.358289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" event={"ID":"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85","Type":"ContainerStarted","Data":"74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290"}
Sep 09 20:21:25.358683 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:25.358422    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:25.377338 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:25.377278    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podStartSLOduration=1.841979027 podStartE2EDuration="25.377257426s" totalImagesPullingTime="19.898528714s" totalInitContainerRuntime="3.636749685s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:25.375239299 +0000 UTC m=+619.751366057" watchObservedRunningTime="2026-09-09 20:21:25.377257426 +0000 UTC m=+619.753384175"
Sep 09 20:21:26.361545 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:26.361513    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:26.362646 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:26.362616    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 09 20:21:27.364464 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:27.364358    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 09 20:21:32.368992 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:32.368957    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:21:32.369762 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:32.369638    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 09 20:21:42.369634 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:42.369584    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 09 20:21:52.370336 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:21:52.370293    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 09 20:22:02.370274 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:22:02.370225    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 09 20:22:12.369926 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:22:12.369880    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 09 20:22:22.370434 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:22:22.370391    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 09 20:22:32.369679 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:22:32.369637    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 09 20:22:42.370903 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:22:42.370873    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:22:50.620661 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:22:50.620624    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:05.680145 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.680088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:05.776826 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.776783    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:05.780201 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.780178    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"]
Sep 09 20:23:05.783289 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.783274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.785781 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.785759    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-f811d-predictor-serving-cert\""
Sep 09 20:23:05.785870 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.785799    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-f811d-kube-rbac-proxy-sar-config\""
Sep 09 20:23:05.803030 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.803005    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"]
Sep 09 20:23:05.853704 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.853677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-cabundle-cert\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.853848 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.853715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.853848 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.853770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtv8\" (UniqueName: \"kubernetes.io/projected/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-api-access-tmtv8\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.853848 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.853806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70282371-edec-45aa-ac9a-5ee5c830ded1-proxy-tls\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.853954 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.853859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70282371-edec-45aa-ac9a-5ee5c830ded1-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.909252 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.909225    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:05.933863 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.933785    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"]
Sep 09 20:23:05.936841 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.936826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:05.939054 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.939034    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-f811d-kube-rbac-proxy-sar-config\""
Sep 09 20:23:05.939194 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.939060    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-f811d-predictor-serving-cert\""
Sep 09 20:23:05.949683 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.949662    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"]
Sep 09 20:23:05.954321 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.954295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.954421 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.954341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtv8\" (UniqueName: \"kubernetes.io/projected/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-api-access-tmtv8\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.954421 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.954365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70282371-edec-45aa-ac9a-5ee5c830ded1-proxy-tls\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.954421 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.954402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70282371-edec-45aa-ac9a-5ee5c830ded1-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.954599 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.954431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-cabundle-cert\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.954814 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.954790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70282371-edec-45aa-ac9a-5ee5c830ded1-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.955013 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.954993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.955116 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.955098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-cabundle-cert\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.957014 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.956991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70282371-edec-45aa-ac9a-5ee5c830ded1-proxy-tls\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:05.961737 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:05.961719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtv8\" (UniqueName: \"kubernetes.io/projected/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-api-access-tmtv8\") pod \"isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:06.005728 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.005698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:06.055332 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.055293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.055332 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.055333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.055528 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.055350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.055528 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.055417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm2xq\" (UniqueName: \"kubernetes.io/projected/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-api-access-zm2xq\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.055528 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.055450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-proxy-tls\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.092493 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.092464    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:06.155897 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.155870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.156057 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.155908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.156057 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.155956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm2xq\" (UniqueName: \"kubernetes.io/projected/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-api-access-zm2xq\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.156057 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.155984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-proxy-tls\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.156057 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.156042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.156557 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.156535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.156848 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.156825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-cabundle-cert\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.157822 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.157800    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:06.157926 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.157861    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-f811d-kube-rbac-proxy-sar-config\""
Sep 09 20:23:06.158117 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.158101    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-f811d-predictor-serving-cert\""
Sep 09 20:23:06.164442 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.164421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm2xq\" (UniqueName: \"kubernetes.io/projected/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-api-access-zm2xq\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.167547 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.167506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.169208 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.169187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-proxy-tls\") pod \"isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.246004 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.245921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:06.362486 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.362426    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"]
Sep 09 20:23:06.364478 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:23:06.364449    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70282371_edec_45aa_ac9a_5ee5c830ded1.slice/crio-ba46b6be68f7a7c73cef464801b65d01dd25d160d26c0f1a6d0184007f845da7 WatchSource:0}: Error finding container ba46b6be68f7a7c73cef464801b65d01dd25d160d26c0f1a6d0184007f845da7: Status 404 returned error can't find the container with id ba46b6be68f7a7c73cef464801b65d01dd25d160d26c0f1a6d0184007f845da7
Sep 09 20:23:06.517269 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.517229    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"]
Sep 09 20:23:06.519740 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:23:06.519717    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d23ed3d_d3c5_40be_8c6e_4544e5536d1d.slice/crio-e76649110ab40cdab442ad88ecfc841b60fe74ec1d64821b54c07abab71c144d WatchSource:0}: Error finding container e76649110ab40cdab442ad88ecfc841b60fe74ec1d64821b54c07abab71c144d: Status 404 returned error can't find the container with id e76649110ab40cdab442ad88ecfc841b60fe74ec1d64821b54c07abab71c144d
Sep 09 20:23:06.697303 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.697260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" event={"ID":"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d","Type":"ContainerStarted","Data":"afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556"}
Sep 09 20:23:06.697693 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.697338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" event={"ID":"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d","Type":"ContainerStarted","Data":"e76649110ab40cdab442ad88ecfc841b60fe74ec1d64821b54c07abab71c144d"}
Sep 09 20:23:06.699038 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.699013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" event={"ID":"70282371-edec-45aa-ac9a-5ee5c830ded1","Type":"ContainerStarted","Data":"d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1"}
Sep 09 20:23:06.699188 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:06.699069    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" event={"ID":"70282371-edec-45aa-ac9a-5ee5c830ded1","Type":"ContainerStarted","Data":"ba46b6be68f7a7c73cef464801b65d01dd25d160d26c0f1a6d0184007f845da7"}
Sep 09 20:23:10.713612 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:10.713583    2579 generic.go:356] "Generic (PLEG): container finished" podID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerID="afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556" exitCode=0
Sep 09 20:23:10.714051 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:10.713656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" event={"ID":"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d","Type":"ContainerDied","Data":"afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556"}
Sep 09 20:23:10.715094 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:10.715075    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:23:10.715280 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:10.715261    2579 generic.go:356] "Generic (PLEG): container finished" podID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerID="d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1" exitCode=0
Sep 09 20:23:10.715348 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:10.715333    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" event={"ID":"70282371-edec-45aa-ac9a-5ee5c830ded1","Type":"ContainerDied","Data":"d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1"}
Sep 09 20:23:11.722711 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:11.722673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" event={"ID":"70282371-edec-45aa-ac9a-5ee5c830ded1","Type":"ContainerStarted","Data":"560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3"}
Sep 09 20:23:11.723189 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:11.722748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" event={"ID":"70282371-edec-45aa-ac9a-5ee5c830ded1","Type":"ContainerStarted","Data":"b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b"}
Sep 09 20:23:11.723189 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:11.723064    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:11.723341 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:11.723234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:11.724979 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:11.724944    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 09 20:23:11.740891 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:11.740852    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podStartSLOduration=2.71727127 podStartE2EDuration="6.740838703s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.023567433s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:11.740067446 +0000 UTC m=+726.116194216" watchObservedRunningTime="2026-09-09 20:23:11.740838703 +0000 UTC m=+726.116965442"
Sep 09 20:23:12.728531 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:12.728489    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 09 20:23:17.732971 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:17.732942    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:23:17.733578 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:17.733541    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 09 20:23:27.733503 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:27.733462    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 09 20:23:28.794342 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:28.794308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" event={"ID":"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d","Type":"ContainerStarted","Data":"39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b"}
Sep 09 20:23:28.794702 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:28.794398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" event={"ID":"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d","Type":"ContainerStarted","Data":"aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063"}
Sep 09 20:23:28.794777 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:28.794698    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:28.794777 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:28.794758    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:28.796049 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:28.796023    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 09 20:23:28.813563 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:28.813485    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" podStartSLOduration=2.197613379 podStartE2EDuration="23.813471863s" totalImagesPullingTime="17.833371991s" totalInitContainerRuntime="3.782486493s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:28.812877158 +0000 UTC m=+743.189003904" watchObservedRunningTime="2026-09-09 20:23:28.813471863 +0000 UTC m=+743.189598649"
Sep 09 20:23:29.798400 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:29.798357    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 09 20:23:34.802851 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:34.802825    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:23:34.803322 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:34.803296    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 09 20:23:37.733431 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:37.733389    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 09 20:23:44.803548 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:44.803509    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 09 20:23:47.733483 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:47.733445    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 09 20:23:50.946150 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:50.946092    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:23:54.804114 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:54.804073    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 09 20:23:57.734454 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:23:57.734414    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 09 20:24:04.803595 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:04.803551    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 09 20:24:07.733555 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:07.733514    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 09 20:24:14.803603 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:14.803559    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 09 20:24:17.734450 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:17.734413    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 09 20:24:20.701923 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:20.701891    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"]
Sep 09 20:24:20.702366 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:20.702142    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container" containerID="cri-o://d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c" gracePeriod=30
Sep 09 20:24:20.702366 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:20.702185    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kube-rbac-proxy" containerID="cri-o://74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290" gracePeriod=30
Sep 09 20:24:20.729706 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:20.729680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:20.981731 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:20.981655    2579 generic.go:356] "Generic (PLEG): container finished" podID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerID="74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290" exitCode=2
Sep 09 20:24:20.981864 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:20.981723    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" event={"ID":"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85","Type":"ContainerDied","Data":"74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290"}
Sep 09 20:24:22.365350 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:22.365301    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Sep 09 20:24:22.369611 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:22.369587    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Sep 09 20:24:23.112624 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:23.112585    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 09 20:24:23.455599 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:23.455554    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:24.804324 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:24.804293    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:24:25.233500 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.233477    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:24:25.363571 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.363480    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-proxy-tls\") pod \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") "
Sep 09 20:24:25.363571 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.363531    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx5kn\" (UniqueName: \"kubernetes.io/projected/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-api-access-kx5kn\") pod \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") "
Sep 09 20:24:25.363571 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.363558    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-rbac-proxy-sar-config\") pod \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") "
Sep 09 20:24:25.363844 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.363590    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kserve-provision-location\") pod \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") "
Sep 09 20:24:25.363844 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.363608    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-cabundle-cert\") pod \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\" (UID: \"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85\") "
Sep 09 20:24:25.363995 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.363968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" (UID: "99f8e0bc-ed07-40f9-a1a1-33f2b6326a85"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:25.364047 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.363967    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" (UID: "99f8e0bc-ed07-40f9-a1a1-33f2b6326a85"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:25.364047 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.364017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" (UID: "99f8e0bc-ed07-40f9-a1a1-33f2b6326a85"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:25.366014 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.365985    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" (UID: "99f8e0bc-ed07-40f9-a1a1-33f2b6326a85"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:24:25.366014 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.366004    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-api-access-kx5kn" (OuterVolumeSpecName: "kube-api-access-kx5kn") pod "99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" (UID: "99f8e0bc-ed07-40f9-a1a1-33f2b6326a85"). InnerVolumeSpecName "kube-api-access-kx5kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:25.464289 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.464242    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:24:25.464289 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.464279    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:24:25.464289 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.464289    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx5kn\" (UniqueName: \"kubernetes.io/projected/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-api-access-kx5kn\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:24:25.464289 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.464298    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:24:25.464289 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.464306    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:24:25.877354 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.877318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:25.943030 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.942998    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:25.986645 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.986621    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"]
Sep 09 20:24:25.986890 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.986878    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="storage-initializer"
Sep 09 20:24:25.986936 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.986891    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="storage-initializer"
Sep 09 20:24:25.986936 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.986898    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container"
Sep 09 20:24:25.986936 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.986903    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container"
Sep 09 20:24:25.986936 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.986910    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kube-rbac-proxy"
Sep 09 20:24:25.986936 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.986915    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kube-rbac-proxy"
Sep 09 20:24:25.987093 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.986954    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kserve-container"
Sep 09 20:24:25.987093 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.986965    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerName="kube-rbac-proxy"
Sep 09 20:24:25.989074 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.989059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:25.991412 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.991393    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 09 20:24:25.991521 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.991424    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 09 20:24:25.999047 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:25.999028    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"]
Sep 09 20:24:26.000877 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.000856    2579 generic.go:356] "Generic (PLEG): container finished" podID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" containerID="d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c" exitCode=0
Sep 09 20:24:26.000962 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.000907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" event={"ID":"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85","Type":"ContainerDied","Data":"d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c"}
Sep 09 20:24:26.000962 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.000923    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"
Sep 09 20:24:26.001048 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.000963    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p" event={"ID":"99f8e0bc-ed07-40f9-a1a1-33f2b6326a85","Type":"ContainerDied","Data":"76011cb828ddb0175cb521d5fc8854fbc7eb4b2a6598628d5923987939222d50"}
Sep 09 20:24:26.001048 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.001003    2579 scope.go:117] "RemoveContainer" containerID="74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290"
Sep 09 20:24:26.008793 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.008740    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:26.009625 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.009596    2579 scope.go:117] "RemoveContainer" containerID="d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c"
Sep 09 20:24:26.017369 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.017353    2579 scope.go:117] "RemoveContainer" containerID="56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a"
Sep 09 20:24:26.024214 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.024192    2579 scope.go:117] "RemoveContainer" containerID="74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290"
Sep 09 20:24:26.024501 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:24:26.024482    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290\": container with ID starting with 74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290 not found: ID does not exist" containerID="74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290"
Sep 09 20:24:26.024555 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.024510    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290"} err="failed to get container status \"74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290\": rpc error: code = NotFound desc = could not find container \"74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290\": container with ID starting with 74649b36f3aa0b8f6a485344808e3b2ec913cd3918a11f8674fa961ea6f76290 not found: ID does not exist"
Sep 09 20:24:26.024555 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.024526    2579 scope.go:117] "RemoveContainer" containerID="d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c"
Sep 09 20:24:26.024768 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:24:26.024750    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c\": container with ID starting with d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c not found: ID does not exist" containerID="d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c"
Sep 09 20:24:26.024814 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.024774    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c"} err="failed to get container status \"d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c\": rpc error: code = NotFound desc = could not find container \"d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c\": container with ID starting with d7a5a1856d41ab78ef9bc4ba83ba805e9b8cc0278250b4f325ecd6a91022c98c not found: ID does not exist"
Sep 09 20:24:26.024814 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.024791    2579 scope.go:117] "RemoveContainer" containerID="56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a"
Sep 09 20:24:26.025009 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:24:26.024993    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a\": container with ID starting with 56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a not found: ID does not exist" containerID="56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a"
Sep 09 20:24:26.025044 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.025015    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a"} err="failed to get container status \"56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a\": rpc error: code = NotFound desc = could not find container \"56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a\": container with ID starting with 56f3c8f5d7b0fcc75c65f55cd2d451e0278b80baeaa6d5e1dc1ee0487dfc8d2a not found: ID does not exist"
Sep 09 20:24:26.028251 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.028231    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"]
Sep 09 20:24:26.032050 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.032032    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6477dbf789-4km2p"]
Sep 09 20:24:26.068086 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.068064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptr6v\" (UniqueName: \"kubernetes.io/projected/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-api-access-ptr6v\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.068190 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.068099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04406c93-4015-4ffb-95af-2c6f85e94c09-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.068190 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.068159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.068256 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.068189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.068256 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.068211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04406c93-4015-4ffb-95af-2c6f85e94c09-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.116028 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.116000    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99f8e0bc-ed07-40f9-a1a1-33f2b6326a85" path="/var/lib/kubelet/pods/99f8e0bc-ed07-40f9-a1a1-33f2b6326a85/volumes"
Sep 09 20:24:26.168676 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.168625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.168676 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.168654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04406c93-4015-4ffb-95af-2c6f85e94c09-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.168816 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.168679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptr6v\" (UniqueName: \"kubernetes.io/projected/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-api-access-ptr6v\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.168816 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.168711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04406c93-4015-4ffb-95af-2c6f85e94c09-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.168816 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.168755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.168816 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:24:26.168792    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 09 20:24:26.169006 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:24:26.168852    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04406c93-4015-4ffb-95af-2c6f85e94c09-proxy-tls podName:04406c93-4015-4ffb-95af-2c6f85e94c09 nodeName:}" failed. No retries permitted until 2026-09-09 20:24:26.668830083 +0000 UTC m=+801.044956825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/04406c93-4015-4ffb-95af-2c6f85e94c09-proxy-tls") pod "isvc-canary-stop-predictor-69f8f5d9cc-rpc42" (UID: "04406c93-4015-4ffb-95af-2c6f85e94c09") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 09 20:24:26.169184 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.169159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04406c93-4015-4ffb-95af-2c6f85e94c09-kserve-provision-location\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.169267 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.169251    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.169344 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.169330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-cabundle-cert\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.178848 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.178828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptr6v\" (UniqueName: \"kubernetes.io/projected/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-api-access-ptr6v\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.382439 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.382410    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:26.671063 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.671025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04406c93-4015-4ffb-95af-2c6f85e94c09-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.673494 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.673466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04406c93-4015-4ffb-95af-2c6f85e94c09-proxy-tls\") pod \"isvc-canary-stop-predictor-69f8f5d9cc-rpc42\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:26.899618 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:26.899580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:27.171911 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:27.171883    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"]
Sep 09 20:24:27.174533 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:24:27.174504    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04406c93_4015_4ffb_95af_2c6f85e94c09.slice/crio-4fd8e32f0f6ca30d79d5f7f45985fe51382a82eba0100afc7306df65ba49fb6b WatchSource:0}: Error finding container 4fd8e32f0f6ca30d79d5f7f45985fe51382a82eba0100afc7306df65ba49fb6b: Status 404 returned error can't find the container with id 4fd8e32f0f6ca30d79d5f7f45985fe51382a82eba0100afc7306df65ba49fb6b
Sep 09 20:24:28.010356 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:28.010324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" event={"ID":"04406c93-4015-4ffb-95af-2c6f85e94c09","Type":"ContainerStarted","Data":"4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043"}
Sep 09 20:24:28.010730 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:28.010380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" event={"ID":"04406c93-4015-4ffb-95af-2c6f85e94c09","Type":"ContainerStarted","Data":"4fd8e32f0f6ca30d79d5f7f45985fe51382a82eba0100afc7306df65ba49fb6b"}
Sep 09 20:24:31.024571 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:31.024538    2579 generic.go:356] "Generic (PLEG): container finished" podID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerID="4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043" exitCode=0
Sep 09 20:24:31.024895 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:31.024577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" event={"ID":"04406c93-4015-4ffb-95af-2c6f85e94c09","Type":"ContainerDied","Data":"4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043"}
Sep 09 20:24:32.030329 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:32.030295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" event={"ID":"04406c93-4015-4ffb-95af-2c6f85e94c09","Type":"ContainerStarted","Data":"3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333"}
Sep 09 20:24:32.030802 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:32.030367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" event={"ID":"04406c93-4015-4ffb-95af-2c6f85e94c09","Type":"ContainerStarted","Data":"5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02"}
Sep 09 20:24:32.030802 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:32.030587    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:32.049673 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:32.049628    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podStartSLOduration=3.311224213 podStartE2EDuration="7.04961466s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.738390447s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:32.047659999 +0000 UTC m=+806.423786758" watchObservedRunningTime="2026-09-09 20:24:32.04961466 +0000 UTC m=+806.425741404"
Sep 09 20:24:33.034194 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:33.034157    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:33.035497 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:33.035469    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 20:24:33.113899 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:33.113865    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:24:34.037385 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:34.037341    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 20:24:39.041450 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:39.041422    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:24:39.041993 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:39.041968    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 20:24:45.816809 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:45.816777    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:45.914707 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:45.914677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:49.041887 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:49.041852    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 20:24:55.888164 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:55.888110    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:55.979699 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:55.979670    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"]
Sep 09 20:24:55.979926 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:55.979898    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container" containerID="cri-o://b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b" gracePeriod=30
Sep 09 20:24:55.979990 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:55.979919    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kube-rbac-proxy" containerID="cri-o://560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3" gracePeriod=30
Sep 09 20:24:55.997694 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:55.997670    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:56.016612 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.016585    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"]
Sep 09 20:24:56.016885 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.016824    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kserve-container" containerID="cri-o://aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063" gracePeriod=30
Sep 09 20:24:56.016988 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.016849    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kube-rbac-proxy" containerID="cri-o://39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b" gracePeriod=30
Sep 09 20:24:56.097523 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.097486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:56.119337 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.119309    2579 generic.go:356] "Generic (PLEG): container finished" podID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerID="560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3" exitCode=2
Sep 09 20:24:56.119477 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.119381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" event={"ID":"70282371-edec-45aa-ac9a-5ee5c830ded1","Type":"ContainerDied","Data":"560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3"}
Sep 09 20:24:56.196831 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.196750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:56.211682 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.211655    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"]
Sep 09 20:24:56.214864 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.214850    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.217112 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.217079    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-2aad0-predictor-serving-cert\""
Sep 09 20:24:56.217112 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.217096    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-2aad0-kube-rbac-proxy-sar-config\""
Sep 09 20:24:56.226447 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.226427    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"]
Sep 09 20:24:56.257185 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.257155    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:56.306631 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.306599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6jbm\" (UniqueName: \"kubernetes.io/projected/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-api-access-s6jbm\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.306631 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.306633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8ec05be-f716-448f-9ad0-2e018af16f01-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.306816 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.306660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.306816 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.306722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ec05be-f716-448f-9ad0-2e018af16f01-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.306816 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.306765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.368872 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.368840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:56.407739 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.407707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.407875 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.407746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6jbm\" (UniqueName: \"kubernetes.io/projected/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-api-access-s6jbm\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.407875 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.407765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8ec05be-f716-448f-9ad0-2e018af16f01-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.407875 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.407787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.407875 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.407830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ec05be-f716-448f-9ad0-2e018af16f01-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.408259 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.408230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8ec05be-f716-448f-9ad0-2e018af16f01-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.408506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.408484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.408605 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.408562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.410519 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.410498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ec05be-f716-448f-9ad0-2e018af16f01-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.415630 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.415609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6jbm\" (UniqueName: \"kubernetes.io/projected/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-api-access-s6jbm\") pod \"isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.469153 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.469061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:24:56.524431 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.524400    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:24:56.795992 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:56.795965    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"]
Sep 09 20:24:56.798395 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:24:56.798374    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ec05be_f716_448f_9ad0_2e018af16f01.slice/crio-b65678733a86d5ae20e5dbda8c911d06531ce51ddf296bc16dfa6e121313de29 WatchSource:0}: Error finding container b65678733a86d5ae20e5dbda8c911d06531ce51ddf296bc16dfa6e121313de29: Status 404 returned error can't find the container with id b65678733a86d5ae20e5dbda8c911d06531ce51ddf296bc16dfa6e121313de29
Sep 09 20:24:57.125778 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:57.125750    2579 generic.go:356] "Generic (PLEG): container finished" podID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerID="39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b" exitCode=2
Sep 09 20:24:57.126203 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:57.125825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" event={"ID":"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d","Type":"ContainerDied","Data":"39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b"}
Sep 09 20:24:57.127341 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:57.127321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" event={"ID":"c8ec05be-f716-448f-9ad0-2e018af16f01","Type":"ContainerStarted","Data":"c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240"}
Sep 09 20:24:57.127458 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:57.127369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" event={"ID":"c8ec05be-f716-448f-9ad0-2e018af16f01","Type":"ContainerStarted","Data":"b65678733a86d5ae20e5dbda8c911d06531ce51ddf296bc16dfa6e121313de29"}
Sep 09 20:24:57.729438 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:57.729395    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Sep 09 20:24:59.042205 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.042161    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 20:24:59.763700 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.763676    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:24:59.935246 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.935213    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-proxy-tls\") pod \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") "
Sep 09 20:24:59.935246 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.935252    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-rbac-proxy-sar-config\") pod \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") "
Sep 09 20:24:59.935474 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.935299    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-cabundle-cert\") pod \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") "
Sep 09 20:24:59.935474 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.935342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kserve-provision-location\") pod \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") "
Sep 09 20:24:59.935474 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.935372    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm2xq\" (UniqueName: \"kubernetes.io/projected/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-api-access-zm2xq\") pod \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\" (UID: \"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d\") "
Sep 09 20:24:59.935703 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.935675    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" (UID: "0d23ed3d-d3c5-40be-8c6e-4544e5536d1d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:59.935703 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.935689    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" (UID: "0d23ed3d-d3c5-40be-8c6e-4544e5536d1d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:59.935799 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.935683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" (UID: "0d23ed3d-d3c5-40be-8c6e-4544e5536d1d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:24:59.937488 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.937466    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-api-access-zm2xq" (OuterVolumeSpecName: "kube-api-access-zm2xq") pod "0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" (UID: "0d23ed3d-d3c5-40be-8c6e-4544e5536d1d"). InnerVolumeSpecName "kube-api-access-zm2xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:59.937558 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:24:59.937540    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" (UID: "0d23ed3d-d3c5-40be-8c6e-4544e5536d1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:00.036759 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.036729    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:25:00.036759 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.036755    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:25:00.036759 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.036764    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm2xq\" (UniqueName: \"kubernetes.io/projected/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-api-access-zm2xq\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:25:00.036948 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.036775    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:25:00.036948 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.036784    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:25:00.138994 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.138960    2579 generic.go:356] "Generic (PLEG): container finished" podID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerID="aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063" exitCode=0
Sep 09 20:25:00.139368 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.139020    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"
Sep 09 20:25:00.139368 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.139041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" event={"ID":"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d","Type":"ContainerDied","Data":"aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063"}
Sep 09 20:25:00.139368 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.139119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr" event={"ID":"0d23ed3d-d3c5-40be-8c6e-4544e5536d1d","Type":"ContainerDied","Data":"e76649110ab40cdab442ad88ecfc841b60fe74ec1d64821b54c07abab71c144d"}
Sep 09 20:25:00.139368 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.139157    2579 scope.go:117] "RemoveContainer" containerID="39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b"
Sep 09 20:25:00.147240 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.147223    2579 scope.go:117] "RemoveContainer" containerID="aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063"
Sep 09 20:25:00.154486 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.154465    2579 scope.go:117] "RemoveContainer" containerID="afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556"
Sep 09 20:25:00.155685 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.155667    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"]
Sep 09 20:25:00.159409 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.159391    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f811d-predictor-768558d969-qmdpr"]
Sep 09 20:25:00.161716 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.161702    2579 scope.go:117] "RemoveContainer" containerID="39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b"
Sep 09 20:25:00.161980 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:25:00.161961    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b\": container with ID starting with 39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b not found: ID does not exist" containerID="39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b"
Sep 09 20:25:00.162061 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.161986    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b"} err="failed to get container status \"39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b\": rpc error: code = NotFound desc = could not find container \"39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b\": container with ID starting with 39689c4c0abd97b9ab712ebec58bafe550148b478bf6e2a5882cc9e714ceb03b not found: ID does not exist"
Sep 09 20:25:00.162061 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.162002    2579 scope.go:117] "RemoveContainer" containerID="aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063"
Sep 09 20:25:00.162266 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:25:00.162251    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063\": container with ID starting with aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063 not found: ID does not exist" containerID="aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063"
Sep 09 20:25:00.162302 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.162270    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063"} err="failed to get container status \"aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063\": rpc error: code = NotFound desc = could not find container \"aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063\": container with ID starting with aac297127fcf98c7c1b35497c3cdc47072b069928d3185c0c15b5333fe6e0063 not found: ID does not exist"
Sep 09 20:25:00.162302 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.162282    2579 scope.go:117] "RemoveContainer" containerID="afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556"
Sep 09 20:25:00.162528 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:25:00.162515    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556\": container with ID starting with afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556 not found: ID does not exist" containerID="afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556"
Sep 09 20:25:00.162578 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.162530    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556"} err="failed to get container status \"afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556\": rpc error: code = NotFound desc = could not find container \"afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556\": container with ID starting with afa894bdb5ab122ce39dc89a1c7cfa17ba1e49d39f4fe6c90e6085b60eaf4556 not found: ID does not exist"
Sep 09 20:25:00.628919 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.628898    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:25:00.742275 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.742207    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-rbac-proxy-sar-config\") pod \"70282371-edec-45aa-ac9a-5ee5c830ded1\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") "
Sep 09 20:25:00.742275 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.742247    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmtv8\" (UniqueName: \"kubernetes.io/projected/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-api-access-tmtv8\") pod \"70282371-edec-45aa-ac9a-5ee5c830ded1\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") "
Sep 09 20:25:00.742275 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.742270    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70282371-edec-45aa-ac9a-5ee5c830ded1-kserve-provision-location\") pod \"70282371-edec-45aa-ac9a-5ee5c830ded1\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") "
Sep 09 20:25:00.742451 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.742307    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70282371-edec-45aa-ac9a-5ee5c830ded1-proxy-tls\") pod \"70282371-edec-45aa-ac9a-5ee5c830ded1\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") "
Sep 09 20:25:00.742451 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.742327    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-cabundle-cert\") pod \"70282371-edec-45aa-ac9a-5ee5c830ded1\" (UID: \"70282371-edec-45aa-ac9a-5ee5c830ded1\") "
Sep 09 20:25:00.742615 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.742587    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "70282371-edec-45aa-ac9a-5ee5c830ded1" (UID: "70282371-edec-45aa-ac9a-5ee5c830ded1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:00.742683 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.742620    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70282371-edec-45aa-ac9a-5ee5c830ded1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70282371-edec-45aa-ac9a-5ee5c830ded1" (UID: "70282371-edec-45aa-ac9a-5ee5c830ded1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:00.742737 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.742718    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70282371-edec-45aa-ac9a-5ee5c830ded1" (UID: "70282371-edec-45aa-ac9a-5ee5c830ded1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:25:00.744368 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.744347    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-api-access-tmtv8" (OuterVolumeSpecName: "kube-api-access-tmtv8") pod "70282371-edec-45aa-ac9a-5ee5c830ded1" (UID: "70282371-edec-45aa-ac9a-5ee5c830ded1"). InnerVolumeSpecName "kube-api-access-tmtv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:00.744524 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.744508    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70282371-edec-45aa-ac9a-5ee5c830ded1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70282371-edec-45aa-ac9a-5ee5c830ded1" (UID: "70282371-edec-45aa-ac9a-5ee5c830ded1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:00.843076 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.843052    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70282371-edec-45aa-ac9a-5ee5c830ded1-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:25:00.843076 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.843074    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70282371-edec-45aa-ac9a-5ee5c830ded1-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:25:00.843076 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.843083    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:25:00.843279 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.843091    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:25:00.843279 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:00.843100    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmtv8\" (UniqueName: \"kubernetes.io/projected/70282371-edec-45aa-ac9a-5ee5c830ded1-kube-api-access-tmtv8\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:25:01.144861 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.144827    2579 generic.go:356] "Generic (PLEG): container finished" podID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerID="b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b" exitCode=0
Sep 09 20:25:01.145335 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.144883    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"
Sep 09 20:25:01.145335 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.144923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" event={"ID":"70282371-edec-45aa-ac9a-5ee5c830ded1","Type":"ContainerDied","Data":"b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b"}
Sep 09 20:25:01.145335 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.144988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5" event={"ID":"70282371-edec-45aa-ac9a-5ee5c830ded1","Type":"ContainerDied","Data":"ba46b6be68f7a7c73cef464801b65d01dd25d160d26c0f1a6d0184007f845da7"}
Sep 09 20:25:01.145335 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.145022    2579 scope.go:117] "RemoveContainer" containerID="560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3"
Sep 09 20:25:01.146736 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.146716    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerID="c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240" exitCode=0
Sep 09 20:25:01.146825 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.146742    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" event={"ID":"c8ec05be-f716-448f-9ad0-2e018af16f01","Type":"ContainerDied","Data":"c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240"}
Sep 09 20:25:01.153510 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.153489    2579 scope.go:117] "RemoveContainer" containerID="b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b"
Sep 09 20:25:01.160652 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.160636    2579 scope.go:117] "RemoveContainer" containerID="d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1"
Sep 09 20:25:01.169693 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.169679    2579 scope.go:117] "RemoveContainer" containerID="560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3"
Sep 09 20:25:01.169979 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:25:01.169960    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3\": container with ID starting with 560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3 not found: ID does not exist" containerID="560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3"
Sep 09 20:25:01.170059 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.169987    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3"} err="failed to get container status \"560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3\": rpc error: code = NotFound desc = could not find container \"560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3\": container with ID starting with 560cbb7dc0f521bbbecff0ebe0ecb8d06cd40e078d291394c0a96fb24fe2d9d3 not found: ID does not exist"
Sep 09 20:25:01.170059 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.170003    2579 scope.go:117] "RemoveContainer" containerID="b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b"
Sep 09 20:25:01.170336 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:25:01.170317    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b\": container with ID starting with b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b not found: ID does not exist" containerID="b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b"
Sep 09 20:25:01.170409 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.170346    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b"} err="failed to get container status \"b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b\": rpc error: code = NotFound desc = could not find container \"b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b\": container with ID starting with b23e1e4b2e3e0eab91a148781f8767dc83f51ab4e99d56c10c63cda69661ae8b not found: ID does not exist"
Sep 09 20:25:01.170409 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.170367    2579 scope.go:117] "RemoveContainer" containerID="d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1"
Sep 09 20:25:01.170647 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:25:01.170631    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1\": container with ID starting with d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1 not found: ID does not exist" containerID="d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1"
Sep 09 20:25:01.170691 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.170652    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1"} err="failed to get container status \"d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1\": rpc error: code = NotFound desc = could not find container \"d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1\": container with ID starting with d99ed320c354f635c461064b8486989c05e8e8b05cff778cdaa835380e57cfc1 not found: ID does not exist"
Sep 09 20:25:01.179706 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.179684    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"]
Sep 09 20:25:01.182449 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:01.182429    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-f811d-predictor-6c68845d44-79cv5"]
Sep 09 20:25:02.116936 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:02.116897    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" path="/var/lib/kubelet/pods/0d23ed3d-d3c5-40be-8c6e-4544e5536d1d/volumes"
Sep 09 20:25:02.117474 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:02.117452    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" path="/var/lib/kubelet/pods/70282371-edec-45aa-ac9a-5ee5c830ded1/volumes"
Sep 09 20:25:02.152931 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:02.152902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" event={"ID":"c8ec05be-f716-448f-9ad0-2e018af16f01","Type":"ContainerStarted","Data":"f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0"}
Sep 09 20:25:02.153360 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:02.152955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" event={"ID":"c8ec05be-f716-448f-9ad0-2e018af16f01","Type":"ContainerStarted","Data":"beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84"}
Sep 09 20:25:02.153360 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:02.153284    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:25:02.153471 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:02.153415    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:25:02.154853 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:02.154829    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 09 20:25:02.171518 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:02.171478    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podStartSLOduration=2.398313564 podStartE2EDuration="6.171466625s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.773153061s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:02.171219879 +0000 UTC m=+836.547346623" watchObservedRunningTime="2026-09-09 20:25:02.171466625 +0000 UTC m=+836.547593369"
Sep 09 20:25:03.156881 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:03.156846    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 09 20:25:04.160447 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:04.160398    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 09 20:25:09.042340 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:09.042302    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 20:25:09.164603 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:09.164577    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:25:09.165037 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:09.165012    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 09 20:25:19.042148 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:19.042098    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 20:25:19.165816 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:19.165772    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 09 20:25:29.042324 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:29.042285    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 20:25:29.164985 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:29.164946    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 09 20:25:39.042689 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:39.042648    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 20:25:39.165854 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:39.165815    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 09 20:25:41.112744 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:41.112698    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Sep 09 20:25:49.165097 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:49.165057    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 09 20:25:51.113726 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:51.113695    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:25:55.951648 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:55.951613    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:55.966074 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:55.966050    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"]
Sep 09 20:25:55.966321 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:55.966298    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container" containerID="cri-o://5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02" gracePeriod=30
Sep 09 20:25:55.966373 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:55.966329    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kube-rbac-proxy" containerID="cri-o://3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333" gracePeriod=30
Sep 09 20:25:56.049135 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:56.049105    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:25:56.347216 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:56.347110    2579 generic.go:356] "Generic (PLEG): container finished" podID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerID="3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333" exitCode=2
Sep 09 20:25:56.347216 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:56.347168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" event={"ID":"04406c93-4015-4ffb-95af-2c6f85e94c09","Type":"ContainerDied","Data":"3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333"}
Sep 09 20:25:59.038510 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:59.038458    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Sep 09 20:25:59.166832 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:25:59.166803    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:26:00.607451 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.607424    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:26:00.657918 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.657859    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04406c93-4015-4ffb-95af-2c6f85e94c09-kserve-provision-location\") pod \"04406c93-4015-4ffb-95af-2c6f85e94c09\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") "
Sep 09 20:26:00.657918 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.657893    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04406c93-4015-4ffb-95af-2c6f85e94c09-proxy-tls\") pod \"04406c93-4015-4ffb-95af-2c6f85e94c09\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") "
Sep 09 20:26:00.658100 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.657921    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-rbac-proxy-sar-config\") pod \"04406c93-4015-4ffb-95af-2c6f85e94c09\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") "
Sep 09 20:26:00.658100 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.657951    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptr6v\" (UniqueName: \"kubernetes.io/projected/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-api-access-ptr6v\") pod \"04406c93-4015-4ffb-95af-2c6f85e94c09\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") "
Sep 09 20:26:00.658100 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.657997    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-cabundle-cert\") pod \"04406c93-4015-4ffb-95af-2c6f85e94c09\" (UID: \"04406c93-4015-4ffb-95af-2c6f85e94c09\") "
Sep 09 20:26:00.658328 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.658290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "04406c93-4015-4ffb-95af-2c6f85e94c09" (UID: "04406c93-4015-4ffb-95af-2c6f85e94c09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:26:00.658328 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.658227    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04406c93-4015-4ffb-95af-2c6f85e94c09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04406c93-4015-4ffb-95af-2c6f85e94c09" (UID: "04406c93-4015-4ffb-95af-2c6f85e94c09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:00.658401 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.658378    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "04406c93-4015-4ffb-95af-2c6f85e94c09" (UID: "04406c93-4015-4ffb-95af-2c6f85e94c09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:26:00.660081 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.660063    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04406c93-4015-4ffb-95af-2c6f85e94c09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04406c93-4015-4ffb-95af-2c6f85e94c09" (UID: "04406c93-4015-4ffb-95af-2c6f85e94c09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:26:00.660183 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.660107    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-api-access-ptr6v" (OuterVolumeSpecName: "kube-api-access-ptr6v") pod "04406c93-4015-4ffb-95af-2c6f85e94c09" (UID: "04406c93-4015-4ffb-95af-2c6f85e94c09"). InnerVolumeSpecName "kube-api-access-ptr6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:00.713389 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.713365    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:00.758542 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.758525    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:26:00.758597 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.758543    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04406c93-4015-4ffb-95af-2c6f85e94c09-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:26:00.758597 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.758552    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04406c93-4015-4ffb-95af-2c6f85e94c09-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:26:00.758597 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.758560    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:26:00.758597 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:00.758568    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptr6v\" (UniqueName: \"kubernetes.io/projected/04406c93-4015-4ffb-95af-2c6f85e94c09-kube-api-access-ptr6v\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:26:01.370865 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.370829    2579 generic.go:356] "Generic (PLEG): container finished" podID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerID="5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02" exitCode=0
Sep 09 20:26:01.371021 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.370886    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"
Sep 09 20:26:01.371021 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.370918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" event={"ID":"04406c93-4015-4ffb-95af-2c6f85e94c09","Type":"ContainerDied","Data":"5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02"}
Sep 09 20:26:01.371021 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.370971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42" event={"ID":"04406c93-4015-4ffb-95af-2c6f85e94c09","Type":"ContainerDied","Data":"4fd8e32f0f6ca30d79d5f7f45985fe51382a82eba0100afc7306df65ba49fb6b"}
Sep 09 20:26:01.371021 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.370997    2579 scope.go:117] "RemoveContainer" containerID="3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333"
Sep 09 20:26:01.379202 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.379184    2579 scope.go:117] "RemoveContainer" containerID="5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02"
Sep 09 20:26:01.386272 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.386248    2579 scope.go:117] "RemoveContainer" containerID="4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043"
Sep 09 20:26:01.393627 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.393606    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"]
Sep 09 20:26:01.394122 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.394104    2579 scope.go:117] "RemoveContainer" containerID="3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333"
Sep 09 20:26:01.394393 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:26:01.394363    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333\": container with ID starting with 3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333 not found: ID does not exist" containerID="3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333"
Sep 09 20:26:01.394437 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.394392    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333"} err="failed to get container status \"3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333\": rpc error: code = NotFound desc = could not find container \"3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333\": container with ID starting with 3d8c7572199295151de356a2a6ea19f59ee444db6538964243faf39192905333 not found: ID does not exist"
Sep 09 20:26:01.394437 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.394414    2579 scope.go:117] "RemoveContainer" containerID="5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02"
Sep 09 20:26:01.394708 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:26:01.394691    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02\": container with ID starting with 5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02 not found: ID does not exist" containerID="5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02"
Sep 09 20:26:01.394749 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.394713    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02"} err="failed to get container status \"5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02\": rpc error: code = NotFound desc = could not find container \"5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02\": container with ID starting with 5c2f22676308317a557ecc8b25ea68eda373dd8ce32eb29c55fc415a2a32cd02 not found: ID does not exist"
Sep 09 20:26:01.394749 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.394727    2579 scope.go:117] "RemoveContainer" containerID="4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043"
Sep 09 20:26:01.394998 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:26:01.394965    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043\": container with ID starting with 4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043 not found: ID does not exist" containerID="4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043"
Sep 09 20:26:01.395052 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.395009    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043"} err="failed to get container status \"4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043\": rpc error: code = NotFound desc = could not find container \"4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043\": container with ID starting with 4f55322a9a6bf6215441c490e803b53dcb46f0dc60ce75f9f37436ed31491043 not found: ID does not exist"
Sep 09 20:26:01.396689 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:01.396666    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-69f8f5d9cc-rpc42"]
Sep 09 20:26:02.116590 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:02.116562    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" path="/var/lib/kubelet/pods/04406c93-4015-4ffb-95af-2c6f85e94c09/volumes"
Sep 09 20:26:06.162107 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.162078    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:06.188649 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188623    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"]
Sep 09 20:26:06.188894 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188881    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="storage-initializer"
Sep 09 20:26:06.188937 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188899    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="storage-initializer"
Sep 09 20:26:06.188937 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188912    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container"
Sep 09 20:26:06.188937 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188918    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container"
Sep 09 20:26:06.188937 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188926    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kube-rbac-proxy"
Sep 09 20:26:06.188937 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188932    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kube-rbac-proxy"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188941    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188946    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188953    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kserve-container"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188958    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kserve-container"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188965    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kube-rbac-proxy"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188970    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kube-rbac-proxy"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188981    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="storage-initializer"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188990    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="storage-initializer"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.188996    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="storage-initializer"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.189001    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="storage-initializer"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.189012    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kube-rbac-proxy"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.189017    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kube-rbac-proxy"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.189055    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kube-rbac-proxy"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.189068    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="70282371-edec-45aa-ac9a-5ee5c830ded1" containerName="kserve-container"
Sep 09 20:26:06.189075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.189075    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kserve-container"
Sep 09 20:26:06.189554 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.189083    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d23ed3d-d3c5-40be-8c6e-4544e5536d1d" containerName="kube-rbac-proxy"
Sep 09 20:26:06.189554 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.189088    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kube-rbac-proxy"
Sep 09 20:26:06.189554 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.189094    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04406c93-4015-4ffb-95af-2c6f85e94c09" containerName="kserve-container"
Sep 09 20:26:06.193247 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.193232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.195723 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.195699    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-0e8b8c-kube-rbac-proxy-sar-config\""
Sep 09 20:26:06.195831 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.195703    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-0e8b8c-predictor-serving-cert\""
Sep 09 20:26:06.203733 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.203710    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"]
Sep 09 20:26:06.299023 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.298992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-cabundle-cert\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.299231 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.299074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0888a80-160c-4283-9b7b-5098359056ba-proxy-tls\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.299231 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.299103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.299231 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.299152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0888a80-160c-4283-9b7b-5098359056ba-kserve-provision-location\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.299231 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.299181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pjnn\" (UniqueName: \"kubernetes.io/projected/a0888a80-160c-4283-9b7b-5098359056ba-kube-api-access-6pjnn\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.400172 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.400077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0888a80-160c-4283-9b7b-5098359056ba-kserve-provision-location\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.400172 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.400108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pjnn\" (UniqueName: \"kubernetes.io/projected/a0888a80-160c-4283-9b7b-5098359056ba-kube-api-access-6pjnn\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.400388 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.400178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-cabundle-cert\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.400388 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.400204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0888a80-160c-4283-9b7b-5098359056ba-proxy-tls\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.400388 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.400223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.400559 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.400523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0888a80-160c-4283-9b7b-5098359056ba-kserve-provision-location\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.400828 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.400809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-kube-rbac-proxy-sar-config\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.400903 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.400886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-cabundle-cert\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.402858 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.402835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0888a80-160c-4283-9b7b-5098359056ba-proxy-tls\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.407816 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.407793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pjnn\" (UniqueName: \"kubernetes.io/projected/a0888a80-160c-4283-9b7b-5098359056ba-kube-api-access-6pjnn\") pod \"isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") " pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.414527 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.414501    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:06.502520 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.502486    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:06.770925 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:06.770899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"]
Sep 09 20:26:06.773318 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:26:06.773290    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0888a80_160c_4283_9b7b_5098359056ba.slice/crio-221a38fd190b23d6ecef4b2a620d82f649e1bee243a22e7d0a3343dc95904fc3 WatchSource:0}: Error finding container 221a38fd190b23d6ecef4b2a620d82f649e1bee243a22e7d0a3343dc95904fc3: Status 404 returned error can't find the container with id 221a38fd190b23d6ecef4b2a620d82f649e1bee243a22e7d0a3343dc95904fc3
Sep 09 20:26:07.400640 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:07.400604    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" event={"ID":"a0888a80-160c-4283-9b7b-5098359056ba","Type":"ContainerStarted","Data":"b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97"}
Sep 09 20:26:07.401013 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:07.400680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" event={"ID":"a0888a80-160c-4283-9b7b-5098359056ba","Type":"ContainerStarted","Data":"221a38fd190b23d6ecef4b2a620d82f649e1bee243a22e7d0a3343dc95904fc3"}
Sep 09 20:26:11.417757 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:11.417720    2579 generic.go:356] "Generic (PLEG): container finished" podID="a0888a80-160c-4283-9b7b-5098359056ba" containerID="b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97" exitCode=0
Sep 09 20:26:11.418184 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:11.417794    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" event={"ID":"a0888a80-160c-4283-9b7b-5098359056ba","Type":"ContainerDied","Data":"b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97"}
Sep 09 20:26:12.427411 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:12.427371    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" event={"ID":"a0888a80-160c-4283-9b7b-5098359056ba","Type":"ContainerStarted","Data":"3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448"}
Sep 09 20:26:12.427896 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:12.427462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" event={"ID":"a0888a80-160c-4283-9b7b-5098359056ba","Type":"ContainerStarted","Data":"aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670"}
Sep 09 20:26:12.427896 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:12.427679    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:12.448635 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:12.448589    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podStartSLOduration=2.63195248 podStartE2EDuration="6.448577139s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.816624659s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:12.448154485 +0000 UTC m=+906.824281229" watchObservedRunningTime="2026-09-09 20:26:12.448577139 +0000 UTC m=+906.824703884"
Sep 09 20:26:13.431456 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:13.431420    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:13.432928 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:13.432900    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:26:14.434955 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:14.434919    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:26:19.439316 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:19.439285    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:26:19.439793 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:19.439770    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:26:29.440254 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:29.440211    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:26:36.116499 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:36.116411    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:37.318231 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:37.318195    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:39.439799 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:39.439764    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:26:46.166249 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:46.166214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:46.274817 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:46.274787    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:46.287787 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:46.287751    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"]
Sep 09 20:26:46.288196 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:46.288146    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container" containerID="cri-o://beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84" gracePeriod=30
Sep 09 20:26:46.288886 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:46.288508    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kube-rbac-proxy" containerID="cri-o://f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0" gracePeriod=30
Sep 09 20:26:46.303553 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:26:46.303525    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ec05be_f716_448f_9ad0_2e018af16f01.slice/crio-f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 20:26:46.344681 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:46.344649    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:46.414067 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:46.414041    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:46.546517 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:46.546432    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerID="f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0" exitCode=2
Sep 09 20:26:46.546517 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:46.546480    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" event={"ID":"c8ec05be-f716-448f-9ad0-2e018af16f01","Type":"ContainerDied","Data":"f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0"}
Sep 09 20:26:46.638147 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:46.638097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:26:49.160929 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:49.160888    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 09 20:26:49.165236 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:49.165214    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 09 20:26:49.440028 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:49.439949    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:26:49.925348 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:49.925320    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:26:50.010655 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.010626    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-cabundle-cert\") pod \"c8ec05be-f716-448f-9ad0-2e018af16f01\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") "
Sep 09 20:26:50.010825 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.010664    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-rbac-proxy-sar-config\") pod \"c8ec05be-f716-448f-9ad0-2e018af16f01\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") "
Sep 09 20:26:50.010825 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.010694    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6jbm\" (UniqueName: \"kubernetes.io/projected/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-api-access-s6jbm\") pod \"c8ec05be-f716-448f-9ad0-2e018af16f01\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") "
Sep 09 20:26:50.010825 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.010727    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ec05be-f716-448f-9ad0-2e018af16f01-proxy-tls\") pod \"c8ec05be-f716-448f-9ad0-2e018af16f01\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") "
Sep 09 20:26:50.010825 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.010752    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8ec05be-f716-448f-9ad0-2e018af16f01-kserve-provision-location\") pod \"c8ec05be-f716-448f-9ad0-2e018af16f01\" (UID: \"c8ec05be-f716-448f-9ad0-2e018af16f01\") "
Sep 09 20:26:50.011039 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.011011    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c8ec05be-f716-448f-9ad0-2e018af16f01" (UID: "c8ec05be-f716-448f-9ad0-2e018af16f01"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:26:50.011096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.011074    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ec05be-f716-448f-9ad0-2e018af16f01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8ec05be-f716-448f-9ad0-2e018af16f01" (UID: "c8ec05be-f716-448f-9ad0-2e018af16f01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:26:50.011096 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.011070    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8ec05be-f716-448f-9ad0-2e018af16f01" (UID: "c8ec05be-f716-448f-9ad0-2e018af16f01"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:26:50.012855 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.012832    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-api-access-s6jbm" (OuterVolumeSpecName: "kube-api-access-s6jbm") pod "c8ec05be-f716-448f-9ad0-2e018af16f01" (UID: "c8ec05be-f716-448f-9ad0-2e018af16f01"). InnerVolumeSpecName "kube-api-access-s6jbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:50.012922 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.012850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ec05be-f716-448f-9ad0-2e018af16f01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8ec05be-f716-448f-9ad0-2e018af16f01" (UID: "c8ec05be-f716-448f-9ad0-2e018af16f01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:26:50.111863 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.111779    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:26:50.111863 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.111809    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6jbm\" (UniqueName: \"kubernetes.io/projected/c8ec05be-f716-448f-9ad0-2e018af16f01-kube-api-access-s6jbm\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:26:50.111863 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.111819    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8ec05be-f716-448f-9ad0-2e018af16f01-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:26:50.111863 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.111828    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8ec05be-f716-448f-9ad0-2e018af16f01-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:26:50.111863 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.111837    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8ec05be-f716-448f-9ad0-2e018af16f01-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:26:50.561703 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.561670    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerID="beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84" exitCode=0
Sep 09 20:26:50.562148 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.561716    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" event={"ID":"c8ec05be-f716-448f-9ad0-2e018af16f01","Type":"ContainerDied","Data":"beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84"}
Sep 09 20:26:50.562148 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.561728    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"
Sep 09 20:26:50.562148 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.561769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc" event={"ID":"c8ec05be-f716-448f-9ad0-2e018af16f01","Type":"ContainerDied","Data":"b65678733a86d5ae20e5dbda8c911d06531ce51ddf296bc16dfa6e121313de29"}
Sep 09 20:26:50.562148 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.561798    2579 scope.go:117] "RemoveContainer" containerID="f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0"
Sep 09 20:26:50.569622 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.569606    2579 scope.go:117] "RemoveContainer" containerID="beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84"
Sep 09 20:26:50.576577 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.576562    2579 scope.go:117] "RemoveContainer" containerID="c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240"
Sep 09 20:26:50.582252 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.582232    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"]
Sep 09 20:26:50.585373 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.585351    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-2aad0-predictor-86cfd698df-4twrc"]
Sep 09 20:26:50.586704 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.586692    2579 scope.go:117] "RemoveContainer" containerID="f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0"
Sep 09 20:26:50.586971 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:26:50.586952    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0\": container with ID starting with f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0 not found: ID does not exist" containerID="f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0"
Sep 09 20:26:50.587033 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.586979    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0"} err="failed to get container status \"f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0\": rpc error: code = NotFound desc = could not find container \"f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0\": container with ID starting with f5e2bcc066a45c1dabd11dc6cbcd7e13a2f5df896fcf4fb8979ffee5d5c38be0 not found: ID does not exist"
Sep 09 20:26:50.587033 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.586995    2579 scope.go:117] "RemoveContainer" containerID="beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84"
Sep 09 20:26:50.587272 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:26:50.587254    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84\": container with ID starting with beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84 not found: ID does not exist" containerID="beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84"
Sep 09 20:26:50.587340 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.587283    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84"} err="failed to get container status \"beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84\": rpc error: code = NotFound desc = could not find container \"beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84\": container with ID starting with beb8ed7e84e01751b8978a8afa37f7e1d8d9db5f48b086621a10cc335c106f84 not found: ID does not exist"
Sep 09 20:26:50.587340 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.587304    2579 scope.go:117] "RemoveContainer" containerID="c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240"
Sep 09 20:26:50.587536 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:26:50.587519    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240\": container with ID starting with c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240 not found: ID does not exist" containerID="c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240"
Sep 09 20:26:50.587585 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:50.587540    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240"} err="failed to get container status \"c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240\": rpc error: code = NotFound desc = could not find container \"c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240\": container with ID starting with c89b299504606c32f43018d477388b321bedbccf26e4f5d134b9398e373ef240 not found: ID does not exist"
Sep 09 20:26:52.116557 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:52.116523    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" path="/var/lib/kubelet/pods/c8ec05be-f716-448f-9ad0-2e018af16f01/volumes"
Sep 09 20:26:59.439749 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:26:59.439715    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:27:06.343182 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:06.343149    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:06.554467 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:06.554359    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:09.440690 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:09.440651    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:27:19.439749 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:19.439715    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 09 20:27:29.441190 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:29.441095    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:27:51.346770 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.346734    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:51.378256 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.378225    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"]
Sep 09 20:27:51.378571 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.378553    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kube-rbac-proxy"
Sep 09 20:27:51.378665 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.378602    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kube-rbac-proxy"
Sep 09 20:27:51.378665 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.378621    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="storage-initializer"
Sep 09 20:27:51.378665 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.378629    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="storage-initializer"
Sep 09 20:27:51.378665 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.378644    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container"
Sep 09 20:27:51.378665 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.378653    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container"
Sep 09 20:27:51.378905 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.378720    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kube-rbac-proxy"
Sep 09 20:27:51.378905 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.378734    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8ec05be-f716-448f-9ad0-2e018af16f01" containerName="kserve-container"
Sep 09 20:27:51.381743 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.381722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.384059 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.384039    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-0e8b8c-kube-rbac-proxy-sar-config\""
Sep 09 20:27:51.384191 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.384063    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-0e8b8c-predictor-serving-cert\""
Sep 09 20:27:51.384191 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.384154    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-0e8b8c-dockercfg-qlxqx\""
Sep 09 20:27:51.384325 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.384309    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-0e8b8c\""
Sep 09 20:27:51.390433 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.390408    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"]
Sep 09 20:27:51.437365 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.437333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.437508 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.437383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kserve-provision-location\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.437508 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.437434    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.437508 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.437469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rp59\" (UniqueName: \"kubernetes.io/projected/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-api-access-2rp59\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.437615 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.437523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-cabundle-cert\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.538078 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.538043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.538078 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.538079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rp59\" (UniqueName: \"kubernetes.io/projected/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-api-access-2rp59\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.538321 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.538105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-cabundle-cert\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.538321 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.538166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.538321 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.538213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kserve-provision-location\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.538443 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:27:51.538381    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-serving-cert: secret "isvc-secondary-0e8b8c-predictor-serving-cert" not found
Sep 09 20:27:51.538486 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:27:51.538477    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls podName:85b8a8ff-060c-4718-971e-e66c7b5c9d06 nodeName:}" failed. No retries permitted until 2026-09-09 20:27:52.03844837 +0000 UTC m=+1006.414575098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls") pod "isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5" (UID: "85b8a8ff-060c-4718-971e-e66c7b5c9d06") : secret "isvc-secondary-0e8b8c-predictor-serving-cert" not found
Sep 09 20:27:51.538654 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.538633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kserve-provision-location\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.538826 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.538809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-cabundle-cert\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.538950 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.538933    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:51.549416 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:51.549387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rp59\" (UniqueName: \"kubernetes.io/projected/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-api-access-2rp59\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:52.042187 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:52.042146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:52.042367 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:27:52.042240    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-serving-cert: secret "isvc-secondary-0e8b8c-predictor-serving-cert" not found
Sep 09 20:27:52.042367 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:27:52.042316    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls podName:85b8a8ff-060c-4718-971e-e66c7b5c9d06 nodeName:}" failed. No retries permitted until 2026-09-09 20:27:53.04230105 +0000 UTC m=+1007.418427777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls") pod "isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5" (UID: "85b8a8ff-060c-4718-971e-e66c7b5c9d06") : secret "isvc-secondary-0e8b8c-predictor-serving-cert" not found
Sep 09 20:27:52.295146 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:52.295042    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:27:53.050273 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:53.050231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:53.052923 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:53.052895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls\") pod \"isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") " pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:53.192258 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:53.192118    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:27:53.465512 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:53.465481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"]
Sep 09 20:27:53.468592 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:27:53.468569    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b8a8ff_060c_4718_971e_e66c7b5c9d06.slice/crio-9670d295ec4d61d926aaa14696d2497eaa6e29ad1b81de5a40b0ea1559a1d318 WatchSource:0}: Error finding container 9670d295ec4d61d926aaa14696d2497eaa6e29ad1b81de5a40b0ea1559a1d318: Status 404 returned error can't find the container with id 9670d295ec4d61d926aaa14696d2497eaa6e29ad1b81de5a40b0ea1559a1d318
Sep 09 20:27:53.784306 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:53.784206    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5" event={"ID":"85b8a8ff-060c-4718-971e-e66c7b5c9d06","Type":"ContainerStarted","Data":"233bcde6570a08b96ae5f60f3a097c1fd7dd48e9598dbc16c0917b58e7230c21"}
Sep 09 20:27:53.784306 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:53.784277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5" event={"ID":"85b8a8ff-060c-4718-971e-e66c7b5c9d06","Type":"ContainerStarted","Data":"9670d295ec4d61d926aaa14696d2497eaa6e29ad1b81de5a40b0ea1559a1d318"}
Sep 09 20:27:58.804261 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:58.804233    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5_85b8a8ff-060c-4718-971e-e66c7b5c9d06/storage-initializer/0.log"
Sep 09 20:27:58.804636 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:58.804273    2579 generic.go:356] "Generic (PLEG): container finished" podID="85b8a8ff-060c-4718-971e-e66c7b5c9d06" containerID="233bcde6570a08b96ae5f60f3a097c1fd7dd48e9598dbc16c0917b58e7230c21" exitCode=1
Sep 09 20:27:58.804636 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:58.804309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5" event={"ID":"85b8a8ff-060c-4718-971e-e66c7b5c9d06","Type":"ContainerDied","Data":"233bcde6570a08b96ae5f60f3a097c1fd7dd48e9598dbc16c0917b58e7230c21"}
Sep 09 20:27:59.809775 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:59.809743    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5_85b8a8ff-060c-4718-971e-e66c7b5c9d06/storage-initializer/0.log"
Sep 09 20:27:59.810255 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:27:59.809806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5" event={"ID":"85b8a8ff-060c-4718-971e-e66c7b5c9d06","Type":"ContainerStarted","Data":"dcac1b17fc94f2f5a7538e856417188dd853ca8f9eeb88ea23b2cbdbfa09fd38"}
Sep 09 20:28:05.832736 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:05.832704    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5_85b8a8ff-060c-4718-971e-e66c7b5c9d06/storage-initializer/1.log"
Sep 09 20:28:05.833168 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:05.833081    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5_85b8a8ff-060c-4718-971e-e66c7b5c9d06/storage-initializer/0.log"
Sep 09 20:28:05.833168 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:05.833110    2579 generic.go:356] "Generic (PLEG): container finished" podID="85b8a8ff-060c-4718-971e-e66c7b5c9d06" containerID="dcac1b17fc94f2f5a7538e856417188dd853ca8f9eeb88ea23b2cbdbfa09fd38" exitCode=1
Sep 09 20:28:05.833277 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:05.833201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5" event={"ID":"85b8a8ff-060c-4718-971e-e66c7b5c9d06","Type":"ContainerDied","Data":"dcac1b17fc94f2f5a7538e856417188dd853ca8f9eeb88ea23b2cbdbfa09fd38"}
Sep 09 20:28:05.833337 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:05.833291    2579 scope.go:117] "RemoveContainer" containerID="233bcde6570a08b96ae5f60f3a097c1fd7dd48e9598dbc16c0917b58e7230c21"
Sep 09 20:28:05.833657 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:05.833641    2579 scope.go:117] "RemoveContainer" containerID="233bcde6570a08b96ae5f60f3a097c1fd7dd48e9598dbc16c0917b58e7230c21"
Sep 09 20:28:05.843270 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:28:05.843245    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5_kserve-ci-e2e-test(85b8a8ff-060c-4718-971e-e66c7b5c9d06)\"" pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5" podUID="85b8a8ff-060c-4718-971e-e66c7b5c9d06"
Sep 09 20:28:05.843729 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:28:05.843701    2579 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5_kserve-ci-e2e-test_85b8a8ff-060c-4718-971e-e66c7b5c9d06_0 in pod sandbox 9670d295ec4d61d926aaa14696d2497eaa6e29ad1b81de5a40b0ea1559a1d318 from index: no such id: '233bcde6570a08b96ae5f60f3a097c1fd7dd48e9598dbc16c0917b58e7230c21'" containerID="233bcde6570a08b96ae5f60f3a097c1fd7dd48e9598dbc16c0917b58e7230c21"
Sep 09 20:28:05.843833 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:05.843735    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233bcde6570a08b96ae5f60f3a097c1fd7dd48e9598dbc16c0917b58e7230c21"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5_kserve-ci-e2e-test_85b8a8ff-060c-4718-971e-e66c7b5c9d06_0 in pod sandbox 9670d295ec4d61d926aaa14696d2497eaa6e29ad1b81de5a40b0ea1559a1d318 from index: no such id: '233bcde6570a08b96ae5f60f3a097c1fd7dd48e9598dbc16c0917b58e7230c21'"
Sep 09 20:28:06.838412 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:06.838384    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5_85b8a8ff-060c-4718-971e-e66c7b5c9d06/storage-initializer/1.log"
Sep 09 20:28:09.407314 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.407282    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"]
Sep 09 20:28:09.428538 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.428505    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"]
Sep 09 20:28:09.428881 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.428803    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container" containerID="cri-o://aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670" gracePeriod=30
Sep 09 20:28:09.428881 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.428827    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kube-rbac-proxy" containerID="cri-o://3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448" gracePeriod=30
Sep 09 20:28:09.434247 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.434207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:09.436357 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.436247    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 09 20:28:09.489992 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.489957    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:09.548974 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.548955    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5_85b8a8ff-060c-4718-971e-e66c7b5c9d06/storage-initializer/1.log"
Sep 09 20:28:09.550604 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.550587    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:28:09.675036 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.674955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-rbac-proxy-sar-config\") pod \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") "
Sep 09 20:28:09.675036 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.674998    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-cabundle-cert\") pod \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") "
Sep 09 20:28:09.675036 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.675017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls\") pod \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") "
Sep 09 20:28:09.675302 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.675059    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kserve-provision-location\") pod \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") "
Sep 09 20:28:09.675302 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.675157    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rp59\" (UniqueName: \"kubernetes.io/projected/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-api-access-2rp59\") pod \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\" (UID: \"85b8a8ff-060c-4718-971e-e66c7b5c9d06\") "
Sep 09 20:28:09.675429 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.675397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85b8a8ff-060c-4718-971e-e66c7b5c9d06" (UID: "85b8a8ff-060c-4718-971e-e66c7b5c9d06"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:09.675429 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.675411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85b8a8ff-060c-4718-971e-e66c7b5c9d06" (UID: "85b8a8ff-060c-4718-971e-e66c7b5c9d06"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:09.675549 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.675426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85b8a8ff-060c-4718-971e-e66c7b5c9d06" (UID: "85b8a8ff-060c-4718-971e-e66c7b5c9d06"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:09.677441 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.677417    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-api-access-2rp59" (OuterVolumeSpecName: "kube-api-access-2rp59") pod "85b8a8ff-060c-4718-971e-e66c7b5c9d06" (UID: "85b8a8ff-060c-4718-971e-e66c7b5c9d06"). InnerVolumeSpecName "kube-api-access-2rp59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:28:09.677525 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.677440    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85b8a8ff-060c-4718-971e-e66c7b5c9d06" (UID: "85b8a8ff-060c-4718-971e-e66c7b5c9d06"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:28:09.775881 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.775849    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rp59\" (UniqueName: \"kubernetes.io/projected/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-api-access-2rp59\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.775881 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.775876    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.775881 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.775886    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85b8a8ff-060c-4718-971e-e66c7b5c9d06-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.776097 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.775895    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85b8a8ff-060c-4718-971e-e66c7b5c9d06-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.776097 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.775903    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85b8a8ff-060c-4718-971e-e66c7b5c9d06-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:09.850668 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.850638    2579 generic.go:356] "Generic (PLEG): container finished" podID="a0888a80-160c-4283-9b7b-5098359056ba" containerID="3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448" exitCode=2
Sep 09 20:28:09.850781 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.850710    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" event={"ID":"a0888a80-160c-4283-9b7b-5098359056ba","Type":"ContainerDied","Data":"3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448"}
Sep 09 20:28:09.851848 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.851833    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5_85b8a8ff-060c-4718-971e-e66c7b5c9d06/storage-initializer/1.log"
Sep 09 20:28:09.851927 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.851914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5" event={"ID":"85b8a8ff-060c-4718-971e-e66c7b5c9d06","Type":"ContainerDied","Data":"9670d295ec4d61d926aaa14696d2497eaa6e29ad1b81de5a40b0ea1559a1d318"}
Sep 09 20:28:09.851965 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.851959    2579 scope.go:117] "RemoveContainer" containerID="dcac1b17fc94f2f5a7538e856417188dd853ca8f9eeb88ea23b2cbdbfa09fd38"
Sep 09 20:28:09.852001 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.851919    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"
Sep 09 20:28:09.884222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.884196    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"]
Sep 09 20:28:09.889690 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:09.889662    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-0e8b8c-predictor-854d589cc4-thjs5"]
Sep 09 20:28:10.116390 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:10.116361    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85b8a8ff-060c-4718-971e-e66c7b5c9d06" path="/var/lib/kubelet/pods/85b8a8ff-060c-4718-971e-e66c7b5c9d06/volumes"
Sep 09 20:28:14.170302 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.170280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:28:14.308760 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.308666    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0888a80-160c-4283-9b7b-5098359056ba-proxy-tls\") pod \"a0888a80-160c-4283-9b7b-5098359056ba\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") "
Sep 09 20:28:14.308760 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.308707    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pjnn\" (UniqueName: \"kubernetes.io/projected/a0888a80-160c-4283-9b7b-5098359056ba-kube-api-access-6pjnn\") pod \"a0888a80-160c-4283-9b7b-5098359056ba\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") "
Sep 09 20:28:14.308760 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.308748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0888a80-160c-4283-9b7b-5098359056ba-kserve-provision-location\") pod \"a0888a80-160c-4283-9b7b-5098359056ba\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") "
Sep 09 20:28:14.309067 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.308788    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-cabundle-cert\") pod \"a0888a80-160c-4283-9b7b-5098359056ba\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") "
Sep 09 20:28:14.309067 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.308811    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-kube-rbac-proxy-sar-config\") pod \"a0888a80-160c-4283-9b7b-5098359056ba\" (UID: \"a0888a80-160c-4283-9b7b-5098359056ba\") "
Sep 09 20:28:14.309212 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.309113    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0888a80-160c-4283-9b7b-5098359056ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0888a80-160c-4283-9b7b-5098359056ba" (UID: "a0888a80-160c-4283-9b7b-5098359056ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:14.309212 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.309184    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a0888a80-160c-4283-9b7b-5098359056ba" (UID: "a0888a80-160c-4283-9b7b-5098359056ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:14.309291 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.309209    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a0888a80-160c-4283-9b7b-5098359056ba" (UID: "a0888a80-160c-4283-9b7b-5098359056ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:14.311044 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.311018    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0888a80-160c-4283-9b7b-5098359056ba-kube-api-access-6pjnn" (OuterVolumeSpecName: "kube-api-access-6pjnn") pod "a0888a80-160c-4283-9b7b-5098359056ba" (UID: "a0888a80-160c-4283-9b7b-5098359056ba"). InnerVolumeSpecName "kube-api-access-6pjnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:28:14.311180 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.311093    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0888a80-160c-4283-9b7b-5098359056ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0888a80-160c-4283-9b7b-5098359056ba" (UID: "a0888a80-160c-4283-9b7b-5098359056ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:28:14.409301 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.409267    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0888a80-160c-4283-9b7b-5098359056ba-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:14.409301 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.409293    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:14.409301 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.409303    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0888a80-160c-4283-9b7b-5098359056ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:14.409301 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.409312    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0888a80-160c-4283-9b7b-5098359056ba-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:14.409541 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.409320    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pjnn\" (UniqueName: \"kubernetes.io/projected/a0888a80-160c-4283-9b7b-5098359056ba-kube-api-access-6pjnn\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:14.578746 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.578672    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:14.618885 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.618854    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"]
Sep 09 20:28:14.619103 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619092    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kube-rbac-proxy"
Sep 09 20:28:14.619157 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619105    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kube-rbac-proxy"
Sep 09 20:28:14.619157 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619114    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b8a8ff-060c-4718-971e-e66c7b5c9d06" containerName="storage-initializer"
Sep 09 20:28:14.619157 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619120    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b8a8ff-060c-4718-971e-e66c7b5c9d06" containerName="storage-initializer"
Sep 09 20:28:14.619157 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619140    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85b8a8ff-060c-4718-971e-e66c7b5c9d06" containerName="storage-initializer"
Sep 09 20:28:14.619157 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619146    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="85b8a8ff-060c-4718-971e-e66c7b5c9d06" containerName="storage-initializer"
Sep 09 20:28:14.619157 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619154    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container"
Sep 09 20:28:14.619157 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619160    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container"
Sep 09 20:28:14.619362 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619173    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="storage-initializer"
Sep 09 20:28:14.619362 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619178    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="storage-initializer"
Sep 09 20:28:14.619362 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619218    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kube-rbac-proxy"
Sep 09 20:28:14.619362 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619224    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b8a8ff-060c-4718-971e-e66c7b5c9d06" containerName="storage-initializer"
Sep 09 20:28:14.619362 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619230    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="85b8a8ff-060c-4718-971e-e66c7b5c9d06" containerName="storage-initializer"
Sep 09 20:28:14.619362 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.619237    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0888a80-160c-4283-9b7b-5098359056ba" containerName="kserve-container"
Sep 09 20:28:14.622121 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.622105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.624545 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.624517    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-8c572d-predictor-serving-cert\""
Sep 09 20:28:14.624545 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.624521    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-8c572d\""
Sep 09 20:28:14.624722 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.624521    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-8c572d-kube-rbac-proxy-sar-config\""
Sep 09 20:28:14.624722 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.624522    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-8c572d-dockercfg-plg4n\""
Sep 09 20:28:14.633122 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.633097    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"]
Sep 09 20:28:14.681092 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.681048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:14.712033 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.712002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-cabundle-cert\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.712189 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.712052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e004f8f9-fe17-4dfe-9873-358880866850-proxy-tls\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.712189 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.712083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtlrq\" (UniqueName: \"kubernetes.io/projected/e004f8f9-fe17-4dfe-9873-358880866850-kube-api-access-rtlrq\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.712189 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.712117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.712189 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.712165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e004f8f9-fe17-4dfe-9873-358880866850-kserve-provision-location\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.813317 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.813283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e004f8f9-fe17-4dfe-9873-358880866850-proxy-tls\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.813434 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.813318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtlrq\" (UniqueName: \"kubernetes.io/projected/e004f8f9-fe17-4dfe-9873-358880866850-kube-api-access-rtlrq\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.813434 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.813369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.813434 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.813399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e004f8f9-fe17-4dfe-9873-358880866850-kserve-provision-location\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.813434 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.813431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-cabundle-cert\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.813903 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.813881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e004f8f9-fe17-4dfe-9873-358880866850-kserve-provision-location\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.814023 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.814003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.814103 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.814086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-cabundle-cert\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.816082 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.816062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e004f8f9-fe17-4dfe-9873-358880866850-proxy-tls\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.821633 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.821613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtlrq\" (UniqueName: \"kubernetes.io/projected/e004f8f9-fe17-4dfe-9873-358880866850-kube-api-access-rtlrq\") pod \"isvc-init-fail-8c572d-predictor-857c76bf94-drg7r\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") " pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:14.870868 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.870840    2579 generic.go:356] "Generic (PLEG): container finished" podID="a0888a80-160c-4283-9b7b-5098359056ba" containerID="aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670" exitCode=0
Sep 09 20:28:14.870983 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.870898    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"
Sep 09 20:28:14.870983 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.870903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" event={"ID":"a0888a80-160c-4283-9b7b-5098359056ba","Type":"ContainerDied","Data":"aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670"}
Sep 09 20:28:14.870983 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.870971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf" event={"ID":"a0888a80-160c-4283-9b7b-5098359056ba","Type":"ContainerDied","Data":"221a38fd190b23d6ecef4b2a620d82f649e1bee243a22e7d0a3343dc95904fc3"}
Sep 09 20:28:14.871081 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.871010    2579 scope.go:117] "RemoveContainer" containerID="3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448"
Sep 09 20:28:14.879457 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.879441    2579 scope.go:117] "RemoveContainer" containerID="aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670"
Sep 09 20:28:14.886241 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.886222    2579 scope.go:117] "RemoveContainer" containerID="b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97"
Sep 09 20:28:14.892098 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.892079    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"]
Sep 09 20:28:14.895879 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.895857    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-0e8b8c-predictor-64cffb8fdf-l65sf"]
Sep 09 20:28:14.896259 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.896240    2579 scope.go:117] "RemoveContainer" containerID="3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448"
Sep 09 20:28:14.896541 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:28:14.896523    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448\": container with ID starting with 3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448 not found: ID does not exist" containerID="3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448"
Sep 09 20:28:14.896611 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.896549    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448"} err="failed to get container status \"3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448\": rpc error: code = NotFound desc = could not find container \"3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448\": container with ID starting with 3ec0df8edfa1b78687e5adeb21828bd1cd84c88dcefae2d26746ec6db242c448 not found: ID does not exist"
Sep 09 20:28:14.896611 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.896568    2579 scope.go:117] "RemoveContainer" containerID="aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670"
Sep 09 20:28:14.896822 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:28:14.896806    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670\": container with ID starting with aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670 not found: ID does not exist" containerID="aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670"
Sep 09 20:28:14.896865 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.896827    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670"} err="failed to get container status \"aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670\": rpc error: code = NotFound desc = could not find container \"aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670\": container with ID starting with aec15d8b810a23f5bc759fdcc3363afeacb5b032621270a5b1aad8c1fa972670 not found: ID does not exist"
Sep 09 20:28:14.896865 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.896848    2579 scope.go:117] "RemoveContainer" containerID="b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97"
Sep 09 20:28:14.897090 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:28:14.897073    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97\": container with ID starting with b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97 not found: ID does not exist" containerID="b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97"
Sep 09 20:28:14.897204 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.897095    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97"} err="failed to get container status \"b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97\": rpc error: code = NotFound desc = could not find container \"b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97\": container with ID starting with b097a052107bd3b9463fe811d32a6cf4cf27293cf26a26946b165b7d8a03db97 not found: ID does not exist"
Sep 09 20:28:14.932436 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:14.932412    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:15.200741 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:15.200662    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"]
Sep 09 20:28:15.203953 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:28:15.203930    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode004f8f9_fe17_4dfe_9873_358880866850.slice/crio-07db72c45fe58bc7c4d8d3a633b0997599c11bee096a9afe567b8191fc01c49b WatchSource:0}: Error finding container 07db72c45fe58bc7c4d8d3a633b0997599c11bee096a9afe567b8191fc01c49b: Status 404 returned error can't find the container with id 07db72c45fe58bc7c4d8d3a633b0997599c11bee096a9afe567b8191fc01c49b
Sep 09 20:28:15.876856 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:15.876817    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r" event={"ID":"e004f8f9-fe17-4dfe-9873-358880866850","Type":"ContainerStarted","Data":"cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230"}
Sep 09 20:28:15.877028 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:15.876893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r" event={"ID":"e004f8f9-fe17-4dfe-9873-358880866850","Type":"ContainerStarted","Data":"07db72c45fe58bc7c4d8d3a633b0997599c11bee096a9afe567b8191fc01c49b"}
Sep 09 20:28:16.117865 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:16.117837    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0888a80-160c-4283-9b7b-5098359056ba" path="/var/lib/kubelet/pods/a0888a80-160c-4283-9b7b-5098359056ba/volumes"
Sep 09 20:28:17.884942 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:17.884912    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8c572d-predictor-857c76bf94-drg7r_e004f8f9-fe17-4dfe-9873-358880866850/storage-initializer/0.log"
Sep 09 20:28:17.885445 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:17.884958    2579 generic.go:356] "Generic (PLEG): container finished" podID="e004f8f9-fe17-4dfe-9873-358880866850" containerID="cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230" exitCode=1
Sep 09 20:28:17.885445 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:17.884991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r" event={"ID":"e004f8f9-fe17-4dfe-9873-358880866850","Type":"ContainerDied","Data":"cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230"}
Sep 09 20:28:18.890745 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:18.890719    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8c572d-predictor-857c76bf94-drg7r_e004f8f9-fe17-4dfe-9873-358880866850/storage-initializer/0.log"
Sep 09 20:28:18.891121 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:18.890802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r" event={"ID":"e004f8f9-fe17-4dfe-9873-358880866850","Type":"ContainerStarted","Data":"a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a"}
Sep 09 20:28:19.608868 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:19.608835    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"]
Sep 09 20:28:19.622472 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:19.622438    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:19.894297 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:19.894261    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r" podUID="e004f8f9-fe17-4dfe-9873-358880866850" containerName="storage-initializer" containerID="cri-o://a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a" gracePeriod=30
Sep 09 20:28:21.031785 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.031763    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8c572d-predictor-857c76bf94-drg7r_e004f8f9-fe17-4dfe-9873-358880866850/storage-initializer/1.log"
Sep 09 20:28:21.032262 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.032224    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8c572d-predictor-857c76bf94-drg7r_e004f8f9-fe17-4dfe-9873-358880866850/storage-initializer/0.log"
Sep 09 20:28:21.035461 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.035434    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:21.159589 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.159494    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtlrq\" (UniqueName: \"kubernetes.io/projected/e004f8f9-fe17-4dfe-9873-358880866850-kube-api-access-rtlrq\") pod \"e004f8f9-fe17-4dfe-9873-358880866850\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") "
Sep 09 20:28:21.159589 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.159549    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e004f8f9-fe17-4dfe-9873-358880866850-kserve-provision-location\") pod \"e004f8f9-fe17-4dfe-9873-358880866850\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") "
Sep 09 20:28:21.159826 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.159597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-kube-rbac-proxy-sar-config\") pod \"e004f8f9-fe17-4dfe-9873-358880866850\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") "
Sep 09 20:28:21.159826 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.159651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e004f8f9-fe17-4dfe-9873-358880866850-proxy-tls\") pod \"e004f8f9-fe17-4dfe-9873-358880866850\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") "
Sep 09 20:28:21.159826 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.159697    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-cabundle-cert\") pod \"e004f8f9-fe17-4dfe-9873-358880866850\" (UID: \"e004f8f9-fe17-4dfe-9873-358880866850\") "
Sep 09 20:28:21.159971 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.159842    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e004f8f9-fe17-4dfe-9873-358880866850-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e004f8f9-fe17-4dfe-9873-358880866850" (UID: "e004f8f9-fe17-4dfe-9873-358880866850"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:21.160031 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.160009    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e004f8f9-fe17-4dfe-9873-358880866850" (UID: "e004f8f9-fe17-4dfe-9873-358880866850"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:21.160153 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.160104    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e004f8f9-fe17-4dfe-9873-358880866850" (UID: "e004f8f9-fe17-4dfe-9873-358880866850"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:28:21.161812 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.161790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e004f8f9-fe17-4dfe-9873-358880866850-kube-api-access-rtlrq" (OuterVolumeSpecName: "kube-api-access-rtlrq") pod "e004f8f9-fe17-4dfe-9873-358880866850" (UID: "e004f8f9-fe17-4dfe-9873-358880866850"). InnerVolumeSpecName "kube-api-access-rtlrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:28:21.161871 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.161853    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e004f8f9-fe17-4dfe-9873-358880866850-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e004f8f9-fe17-4dfe-9873-358880866850" (UID: "e004f8f9-fe17-4dfe-9873-358880866850"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:28:21.261100 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.261067    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e004f8f9-fe17-4dfe-9873-358880866850-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:21.261100 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.261091    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:21.261100 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.261100    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtlrq\" (UniqueName: \"kubernetes.io/projected/e004f8f9-fe17-4dfe-9873-358880866850-kube-api-access-rtlrq\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:21.261331 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.261110    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e004f8f9-fe17-4dfe-9873-358880866850-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:21.261331 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.261120    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e004f8f9-fe17-4dfe-9873-358880866850-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:28:21.902360 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.902327    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8c572d-predictor-857c76bf94-drg7r_e004f8f9-fe17-4dfe-9873-358880866850/storage-initializer/1.log"
Sep 09 20:28:21.902701 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.902686    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-8c572d-predictor-857c76bf94-drg7r_e004f8f9-fe17-4dfe-9873-358880866850/storage-initializer/0.log"
Sep 09 20:28:21.902815 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.902723    2579 generic.go:356] "Generic (PLEG): container finished" podID="e004f8f9-fe17-4dfe-9873-358880866850" containerID="a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a" exitCode=1
Sep 09 20:28:21.902815 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.902779    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"
Sep 09 20:28:21.902890 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.902781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r" event={"ID":"e004f8f9-fe17-4dfe-9873-358880866850","Type":"ContainerDied","Data":"a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a"}
Sep 09 20:28:21.902929 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.902917    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r" event={"ID":"e004f8f9-fe17-4dfe-9873-358880866850","Type":"ContainerDied","Data":"07db72c45fe58bc7c4d8d3a633b0997599c11bee096a9afe567b8191fc01c49b"}
Sep 09 20:28:21.902971 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.902949    2579 scope.go:117] "RemoveContainer" containerID="a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a"
Sep 09 20:28:21.912012 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.911876    2579 scope.go:117] "RemoveContainer" containerID="cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230"
Sep 09 20:28:21.919475 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.919453    2579 scope.go:117] "RemoveContainer" containerID="a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a"
Sep 09 20:28:21.919764 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:28:21.919745    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a\": container with ID starting with a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a not found: ID does not exist" containerID="a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a"
Sep 09 20:28:21.919826 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.919776    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a"} err="failed to get container status \"a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a\": rpc error: code = NotFound desc = could not find container \"a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a\": container with ID starting with a120eab676d3d833dfd46031bfc6728bdbe0a81050b88498c76569975703477a not found: ID does not exist"
Sep 09 20:28:21.919826 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.919800    2579 scope.go:117] "RemoveContainer" containerID="cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230"
Sep 09 20:28:21.920041 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:28:21.920023    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230\": container with ID starting with cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230 not found: ID does not exist" containerID="cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230"
Sep 09 20:28:21.920077 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.920048    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230"} err="failed to get container status \"cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230\": rpc error: code = NotFound desc = could not find container \"cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230\": container with ID starting with cf23b13f5408e6526ab02becca08342ab64c0b3df9ee2d9c0c592c09c02a6230 not found: ID does not exist"
Sep 09 20:28:21.939739 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.939707    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"]
Sep 09 20:28:21.943174 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:21.943149    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-8c572d-predictor-857c76bf94-drg7r"]
Sep 09 20:28:22.116373 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:22.116327    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e004f8f9-fe17-4dfe-9873-358880866850" path="/var/lib/kubelet/pods/e004f8f9-fe17-4dfe-9873-358880866850/volumes"
Sep 09 20:28:24.863154 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.863107    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:24.892613 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.892579    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"]
Sep 09 20:28:24.892842 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.892830    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e004f8f9-fe17-4dfe-9873-358880866850" containerName="storage-initializer"
Sep 09 20:28:24.892885 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.892844    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e004f8f9-fe17-4dfe-9873-358880866850" containerName="storage-initializer"
Sep 09 20:28:24.892885 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.892853    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e004f8f9-fe17-4dfe-9873-358880866850" containerName="storage-initializer"
Sep 09 20:28:24.892885 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.892858    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e004f8f9-fe17-4dfe-9873-358880866850" containerName="storage-initializer"
Sep 09 20:28:24.893016 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.892899    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e004f8f9-fe17-4dfe-9873-358880866850" containerName="storage-initializer"
Sep 09 20:28:24.893016 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.892906    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e004f8f9-fe17-4dfe-9873-358880866850" containerName="storage-initializer"
Sep 09 20:28:24.897515 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.897498    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:24.899940 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.899917    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 20:28:24.899940 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.899932    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 20:28:24.900219 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.899966    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 20:28:24.900219 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.900092    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-cf411-kube-rbac-proxy-sar-config\""
Sep 09 20:28:24.900331 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.900245    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-cf411-predictor-serving-cert\""
Sep 09 20:28:24.900389 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.900334    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f4cxz\""
Sep 09 20:28:24.900626 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.900613    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 20:28:24.905310 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.905284    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"]
Sep 09 20:28:24.989240 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.989193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7cbcc79-9968-445b-81bc-725757fa5271-kserve-provision-location\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:24.989410 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.989248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7cbcc79-9968-445b-81bc-725757fa5271-proxy-tls\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:24.989410 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.989282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:24.989410 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.989334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpg4n\" (UniqueName: \"kubernetes.io/projected/f7cbcc79-9968-445b-81bc-725757fa5271-kube-api-access-hpg4n\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:24.989515 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:24.989409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-cabundle-cert\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.090330 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.090289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-cabundle-cert\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.090330 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.090329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7cbcc79-9968-445b-81bc-725757fa5271-kserve-provision-location\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.090588 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.090358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7cbcc79-9968-445b-81bc-725757fa5271-proxy-tls\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.090588 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.090388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.090588 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.090404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpg4n\" (UniqueName: \"kubernetes.io/projected/f7cbcc79-9968-445b-81bc-725757fa5271-kube-api-access-hpg4n\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.090785 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.090756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7cbcc79-9968-445b-81bc-725757fa5271-kserve-provision-location\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.091043 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.091024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-cabundle-cert\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.091089 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.091034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.093083 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.093048    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7cbcc79-9968-445b-81bc-725757fa5271-proxy-tls\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.098813 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.098786    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpg4n\" (UniqueName: \"kubernetes.io/projected/f7cbcc79-9968-445b-81bc-725757fa5271-kube-api-access-hpg4n\") pod \"raw-sklearn-cf411-predictor-fd44f859b-7884b\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") " pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.110313 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.110274    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:25.207754 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.207710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:25.478647 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.478608    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"]
Sep 09 20:28:25.481345 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:28:25.481317    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7cbcc79_9968_445b_81bc_725757fa5271.slice/crio-78deef9f42e2dd857e5a7784ddb00ec2e840be4e6ed7dc878c082bbec7ba53c9 WatchSource:0}: Error finding container 78deef9f42e2dd857e5a7784ddb00ec2e840be4e6ed7dc878c082bbec7ba53c9: Status 404 returned error can't find the container with id 78deef9f42e2dd857e5a7784ddb00ec2e840be4e6ed7dc878c082bbec7ba53c9
Sep 09 20:28:25.924827 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.924791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" event={"ID":"f7cbcc79-9968-445b-81bc-725757fa5271","Type":"ContainerStarted","Data":"ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321"}
Sep 09 20:28:25.925357 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:25.924855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" event={"ID":"f7cbcc79-9968-445b-81bc-725757fa5271","Type":"ContainerStarted","Data":"78deef9f42e2dd857e5a7784ddb00ec2e840be4e6ed7dc878c082bbec7ba53c9"}
Sep 09 20:28:29.942590 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:29.942554    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7cbcc79-9968-445b-81bc-725757fa5271" containerID="ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321" exitCode=0
Sep 09 20:28:29.943003 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:29.942618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" event={"ID":"f7cbcc79-9968-445b-81bc-725757fa5271","Type":"ContainerDied","Data":"ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321"}
Sep 09 20:28:30.948005 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:30.947967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" event={"ID":"f7cbcc79-9968-445b-81bc-725757fa5271","Type":"ContainerStarted","Data":"e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2"}
Sep 09 20:28:30.948504 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:30.948039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" event={"ID":"f7cbcc79-9968-445b-81bc-725757fa5271","Type":"ContainerStarted","Data":"6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba"}
Sep 09 20:28:30.948504 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:30.948283    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:30.967540 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:30.967486    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podStartSLOduration=2.990291986 podStartE2EDuration="6.967469565s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.977177579s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:30.965333329 +0000 UTC m=+1045.341460074" watchObservedRunningTime="2026-09-09 20:28:30.967469565 +0000 UTC m=+1045.343596311"
Sep 09 20:28:31.951482 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:31.951449    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:31.952813 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:31.952784    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:28:32.954644 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:32.954608    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:28:37.959825 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:37.959794    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:28:37.960305 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:37.960279    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:28:41.571709 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.571680    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:41.668761 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.668731    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"]
Sep 09 20:28:41.668928 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.668773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:41.671961 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.671947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.674305 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.674280    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-38f5d-predictor-serving-cert\""
Sep 09 20:28:41.674444 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.674346    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-38f5d-kube-rbac-proxy-sar-config\""
Sep 09 20:28:41.681567 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.681546    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"]
Sep 09 20:28:41.710833 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.710810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.710959 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.710846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.710959 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.710866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.710959 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.710884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdhj\" (UniqueName: \"kubernetes.io/projected/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-api-access-fcdhj\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.710959 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.710927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-proxy-tls\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.741753 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.741726    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:41.811860 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.811836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.811966 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.811869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.811966 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.811891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.812059 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.812007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdhj\" (UniqueName: \"kubernetes.io/projected/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-api-access-fcdhj\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.812115 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.812056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-proxy-tls\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.812272 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.812250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.812615 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.812594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-cabundle-cert\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.812685 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.812595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.814558 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.814541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-proxy-tls\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.819520 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.819499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdhj\" (UniqueName: \"kubernetes.io/projected/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-api-access-fcdhj\") pod \"isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:41.845546 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.845495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:41.982289 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:41.982254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:42.251892 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:42.251858    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"]
Sep 09 20:28:42.254703 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:28:42.254669    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8c3144_a6a7_4a98_b9c3_82467f0a4cb3.slice/crio-2e86ddf803834bc16a85f173f7a5eb9cb8b530d35984f5122b0e2eec7b5a66d1 WatchSource:0}: Error finding container 2e86ddf803834bc16a85f173f7a5eb9cb8b530d35984f5122b0e2eec7b5a66d1: Status 404 returned error can't find the container with id 2e86ddf803834bc16a85f173f7a5eb9cb8b530d35984f5122b0e2eec7b5a66d1
Sep 09 20:28:42.273537 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:42.273516    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:42.377403 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:42.377366    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:28:42.990973 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:42.990941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" event={"ID":"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3","Type":"ContainerStarted","Data":"c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76"}
Sep 09 20:28:42.991363 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:42.990995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" event={"ID":"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3","Type":"ContainerStarted","Data":"2e86ddf803834bc16a85f173f7a5eb9cb8b530d35984f5122b0e2eec7b5a66d1"}
Sep 09 20:28:47.006065 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:47.006033    2579 generic.go:356] "Generic (PLEG): container finished" podID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerID="c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76" exitCode=0
Sep 09 20:28:47.006065 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:47.006074    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" event={"ID":"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3","Type":"ContainerDied","Data":"c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76"}
Sep 09 20:28:47.960980 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:47.960937    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:28:48.011464 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:48.011433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" event={"ID":"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3","Type":"ContainerStarted","Data":"b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912"}
Sep 09 20:28:48.011841 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:48.011491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" event={"ID":"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3","Type":"ContainerStarted","Data":"09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3"}
Sep 09 20:28:48.011841 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:48.011800    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:48.011964 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:48.011945    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:48.013377 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:48.013347    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:28:48.028647 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:48.028605    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podStartSLOduration=3.309440554 podStartE2EDuration="7.028593537s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.719152983s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:48.027434903 +0000 UTC m=+1062.403561648" watchObservedRunningTime="2026-09-09 20:28:48.028593537 +0000 UTC m=+1062.404720282"
Sep 09 20:28:49.014951 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:49.014910    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:28:54.019639 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:54.019606    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:28:54.020302 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:54.020270    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:28:57.960562 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:28:57.960477    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:04.020727 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:04.020681    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:29:07.960376 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:07.960331    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:14.021029 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:14.020988    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:29:17.960294 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:17.960253    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:24.020246 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:24.020200    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:29:27.961191 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:27.961150    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:34.020915 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:34.020874    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:29:37.961045 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:37.961005    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:42.112980 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:42.112938    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Sep 09 20:29:44.020339 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:44.020300    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:29:52.116253 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:52.116223    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:29:54.020408 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:54.020362    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:29:54.909336 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:54.909303    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"]
Sep 09 20:29:54.909584 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:54.909535    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container" containerID="cri-o://6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba" gracePeriod=30
Sep 09 20:29:54.909673 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:54.909562    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kube-rbac-proxy" containerID="cri-o://e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2" gracePeriod=30
Sep 09 20:29:54.928465 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:54.928437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:29:55.251976 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:55.251885    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7cbcc79-9968-445b-81bc-725757fa5271" containerID="e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2" exitCode=2
Sep 09 20:29:55.252387 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:55.251969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" event={"ID":"f7cbcc79-9968-445b-81bc-725757fa5271","Type":"ContainerDied","Data":"e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2"}
Sep 09 20:29:57.955584 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:57.955543    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 09 20:29:59.112073 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.112031    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:29:59.549533 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.549512    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:29:59.696430 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.696331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpg4n\" (UniqueName: \"kubernetes.io/projected/f7cbcc79-9968-445b-81bc-725757fa5271-kube-api-access-hpg4n\") pod \"f7cbcc79-9968-445b-81bc-725757fa5271\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") "
Sep 09 20:29:59.696430 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.696376    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-kube-rbac-proxy-sar-config\") pod \"f7cbcc79-9968-445b-81bc-725757fa5271\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") "
Sep 09 20:29:59.696650 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.696442    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7cbcc79-9968-445b-81bc-725757fa5271-kserve-provision-location\") pod \"f7cbcc79-9968-445b-81bc-725757fa5271\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") "
Sep 09 20:29:59.696650 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.696480    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7cbcc79-9968-445b-81bc-725757fa5271-proxy-tls\") pod \"f7cbcc79-9968-445b-81bc-725757fa5271\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") "
Sep 09 20:29:59.696650 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.696497    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-cabundle-cert\") pod \"f7cbcc79-9968-445b-81bc-725757fa5271\" (UID: \"f7cbcc79-9968-445b-81bc-725757fa5271\") "
Sep 09 20:29:59.696821 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.696791    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7cbcc79-9968-445b-81bc-725757fa5271-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7cbcc79-9968-445b-81bc-725757fa5271" (UID: "f7cbcc79-9968-445b-81bc-725757fa5271"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:29:59.696862 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.696816    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7cbcc79-9968-445b-81bc-725757fa5271" (UID: "f7cbcc79-9968-445b-81bc-725757fa5271"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:29:59.696896 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.696856    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7cbcc79-9968-445b-81bc-725757fa5271" (UID: "f7cbcc79-9968-445b-81bc-725757fa5271"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:29:59.698701 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.698678    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7cbcc79-9968-445b-81bc-725757fa5271-kube-api-access-hpg4n" (OuterVolumeSpecName: "kube-api-access-hpg4n") pod "f7cbcc79-9968-445b-81bc-725757fa5271" (UID: "f7cbcc79-9968-445b-81bc-725757fa5271"). InnerVolumeSpecName "kube-api-access-hpg4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:29:59.698818 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.698752    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7cbcc79-9968-445b-81bc-725757fa5271-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7cbcc79-9968-445b-81bc-725757fa5271" (UID: "f7cbcc79-9968-445b-81bc-725757fa5271"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:29:59.797256 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.797221    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7cbcc79-9968-445b-81bc-725757fa5271-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:29:59.797256 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.797253    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7cbcc79-9968-445b-81bc-725757fa5271-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:29:59.797256 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.797262    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:29:59.797427 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.797271    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpg4n\" (UniqueName: \"kubernetes.io/projected/f7cbcc79-9968-445b-81bc-725757fa5271-kube-api-access-hpg4n\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:29:59.797427 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:29:59.797279    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7cbcc79-9968-445b-81bc-725757fa5271-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:30:00.052720 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.052636    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:00.273245 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.273215    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7cbcc79-9968-445b-81bc-725757fa5271" containerID="6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba" exitCode=0
Sep 09 20:30:00.273663 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.273279    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"
Sep 09 20:30:00.273663 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.273310    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" event={"ID":"f7cbcc79-9968-445b-81bc-725757fa5271","Type":"ContainerDied","Data":"6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba"}
Sep 09 20:30:00.273663 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.273378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b" event={"ID":"f7cbcc79-9968-445b-81bc-725757fa5271","Type":"ContainerDied","Data":"78deef9f42e2dd857e5a7784ddb00ec2e840be4e6ed7dc878c082bbec7ba53c9"}
Sep 09 20:30:00.273663 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.273417    2579 scope.go:117] "RemoveContainer" containerID="e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2"
Sep 09 20:30:00.281713 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.281641    2579 scope.go:117] "RemoveContainer" containerID="6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba"
Sep 09 20:30:00.288880 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.288852    2579 scope.go:117] "RemoveContainer" containerID="ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321"
Sep 09 20:30:00.289866 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.289846    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"]
Sep 09 20:30:00.294275 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.294254    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-cf411-predictor-fd44f859b-7884b"]
Sep 09 20:30:00.296433 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.296420    2579 scope.go:117] "RemoveContainer" containerID="e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2"
Sep 09 20:30:00.296683 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:30:00.296666    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2\": container with ID starting with e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2 not found: ID does not exist" containerID="e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2"
Sep 09 20:30:00.296743 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.296690    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2"} err="failed to get container status \"e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2\": rpc error: code = NotFound desc = could not find container \"e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2\": container with ID starting with e53680b6c5a412f2398cf05cbd6000f72e76880b3f05093d632f2125c113a2c2 not found: ID does not exist"
Sep 09 20:30:00.296743 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.296708    2579 scope.go:117] "RemoveContainer" containerID="6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba"
Sep 09 20:30:00.296951 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:30:00.296937    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba\": container with ID starting with 6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba not found: ID does not exist" containerID="6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba"
Sep 09 20:30:00.296994 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.296956    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba"} err="failed to get container status \"6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba\": rpc error: code = NotFound desc = could not find container \"6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba\": container with ID starting with 6151d94d3db9a4ded6a9c3981c61e6faaf1d67a4382968982d36b085cfdaf0ba not found: ID does not exist"
Sep 09 20:30:00.296994 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.296969    2579 scope.go:117] "RemoveContainer" containerID="ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321"
Sep 09 20:30:00.297208 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:30:00.297192    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321\": container with ID starting with ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321 not found: ID does not exist" containerID="ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321"
Sep 09 20:30:00.297274 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.297210    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321"} err="failed to get container status \"ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321\": rpc error: code = NotFound desc = could not find container \"ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321\": container with ID starting with ad63007a3b537f86f1eb00b3e5ecfb440df647436b534a3470511cef4eda2321 not found: ID does not exist"
Sep 09 20:30:00.513427 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:00.513395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:30:02.116629 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:02.116592    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" path="/var/lib/kubelet/pods/f7cbcc79-9968-445b-81bc-725757fa5271/volumes"
Sep 09 20:30:09.112011 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:09.111968    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:30:19.112302 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:19.112255    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:30:29.112178 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:29.112065    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:30:39.112375 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:39.112331    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:30:49.112947 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:49.112903    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:30:59.112021 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:30:59.111975    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:31:09.112614 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:09.112567    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:31:19.112905 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:19.112873    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:31:21.731390 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:21.731357    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"]
Sep 09 20:31:21.731794 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:21.731565    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" containerID="cri-o://09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3" gracePeriod=30
Sep 09 20:31:21.731794 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:21.731589    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kube-rbac-proxy" containerID="cri-o://b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912" gracePeriod=30
Sep 09 20:31:21.751458 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:21.751413    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:22.553285 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:22.553250    2579 generic.go:356] "Generic (PLEG): container finished" podID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerID="b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912" exitCode=2
Sep 09 20:31:22.553453 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:22.553305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" event={"ID":"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3","Type":"ContainerDied","Data":"b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912"}
Sep 09 20:31:24.015345 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:24.015307    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 09 20:31:26.871917 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.871874    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:26.904702 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.904660    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"]
Sep 09 20:31:26.904921 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.904909    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="storage-initializer"
Sep 09 20:31:26.904964 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.904923    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="storage-initializer"
Sep 09 20:31:26.904964 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.904930    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container"
Sep 09 20:31:26.904964 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.904936    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container"
Sep 09 20:31:26.904964 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.904952    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kube-rbac-proxy"
Sep 09 20:31:26.904964 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.904958    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kube-rbac-proxy"
Sep 09 20:31:26.905145 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.904997    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kube-rbac-proxy"
Sep 09 20:31:26.905145 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.905006    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cbcc79-9968-445b-81bc-725757fa5271" containerName="kserve-container"
Sep 09 20:31:26.907834 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.907817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:26.910036 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.910014    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 20:31:26.910180 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.910018    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 09 20:31:26.917443 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.917420    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"]
Sep 09 20:31:26.941109 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.941079    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:26.991540 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.991516    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"]
Sep 09 20:31:26.994476 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.994456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:26.996471 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:26.996453    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 20:31:27.005023 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.005002    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"]
Sep 09 20:31:27.018225 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.018205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:27.020264 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.020246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.020378 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.020295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.020378 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.020366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6qfb\" (UniqueName: \"kubernetes.io/projected/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-api-access-b6qfb\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.020496 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.020473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.020557 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.020505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.120955 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.120925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.121089 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.120964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.121089 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.120985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.121089 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.121004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vh4\" (UniqueName: \"kubernetes.io/projected/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-api-access-64vh4\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.121089 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.121024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a1a317-2751-4ae6-8711-d0806d145ba9-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.121089 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.121049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.121535 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.121098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a1a317-2751-4ae6-8711-d0806d145ba9-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.121535 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.121143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.121535 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.121180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.121535 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.121206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6qfb\" (UniqueName: \"kubernetes.io/projected/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-api-access-b6qfb\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.121535 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.121399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.121736 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.121670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.121736 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.121694    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.123702 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.123661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.129112 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.129087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6qfb\" (UniqueName: \"kubernetes.io/projected/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-api-access-b6qfb\") pod \"isvc-canary-create-v2-predictor-559cdfb684-42k74\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.135616 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.135596    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:27.217712 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.217682    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:27.222017 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.221987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.222082 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.222038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vh4\" (UniqueName: \"kubernetes.io/projected/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-api-access-64vh4\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.222082 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.222073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a1a317-2751-4ae6-8711-d0806d145ba9-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.222195 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.222121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a1a317-2751-4ae6-8711-d0806d145ba9-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.222289 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.222266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.222721 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.222572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a1a317-2751-4ae6-8711-d0806d145ba9-kserve-provision-location\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.228104 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.223103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-cabundle-cert\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.228104 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.223334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.228807 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.228789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a1a317-2751-4ae6-8711-d0806d145ba9-proxy-tls\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.233340 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.233317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vh4\" (UniqueName: \"kubernetes.io/projected/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-api-access-64vh4\") pod \"isvc-canary-create-predictor-5f7b4999bf-jg2xb\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.303620 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.303580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:27.511527 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.511445    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"]
Sep 09 20:31:27.514873 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:31:27.514833    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2c60ca_0045_4948_a75a_3aafa3eca2f2.slice/crio-a262bb2f0043e55409d1492dc8010c54f66d47be4f96368d8e9b8c3d9e5c3d72 WatchSource:0}: Error finding container a262bb2f0043e55409d1492dc8010c54f66d47be4f96368d8e9b8c3d9e5c3d72: Status 404 returned error can't find the container with id a262bb2f0043e55409d1492dc8010c54f66d47be4f96368d8e9b8c3d9e5c3d72
Sep 09 20:31:27.571958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.571924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" event={"ID":"ce2c60ca-0045-4948-a75a-3aafa3eca2f2","Type":"ContainerStarted","Data":"d7f62b3ca2c7ca752f90d91d6b1325c783e71c1e4540c345f4089638f81da329"}
Sep 09 20:31:27.572057 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.571999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" event={"ID":"ce2c60ca-0045-4948-a75a-3aafa3eca2f2","Type":"ContainerStarted","Data":"a262bb2f0043e55409d1492dc8010c54f66d47be4f96368d8e9b8c3d9e5c3d72"}
Sep 09 20:31:27.585679 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:27.585656    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"]
Sep 09 20:31:27.587604 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:31:27.587576    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a1a317_2751_4ae6_8711_d0806d145ba9.slice/crio-1b561749e441a29da77a61fd629da5fc575a10a534ad66c6d64be2861867bf98 WatchSource:0}: Error finding container 1b561749e441a29da77a61fd629da5fc575a10a534ad66c6d64be2861867bf98: Status 404 returned error can't find the container with id 1b561749e441a29da77a61fd629da5fc575a10a534ad66c6d64be2861867bf98
Sep 09 20:31:28.577834 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:28.577786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" event={"ID":"29a1a317-2751-4ae6-8711-d0806d145ba9","Type":"ContainerStarted","Data":"c4d4406dfe1a3d62ce32050c5c6436658a5176dce8dcbb30b39534a5674c9fb7"}
Sep 09 20:31:28.578244 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:28.577904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" event={"ID":"29a1a317-2751-4ae6-8711-d0806d145ba9","Type":"ContainerStarted","Data":"1b561749e441a29da77a61fd629da5fc575a10a534ad66c6d64be2861867bf98"}
Sep 09 20:31:29.015364 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:29.015317    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Sep 09 20:31:29.112400 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:29.112351    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 09 20:31:30.198326 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:30.198293    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:31:31.482557 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.482535    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:31:31.589585 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.589558    2579 generic.go:356] "Generic (PLEG): container finished" podID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerID="c4d4406dfe1a3d62ce32050c5c6436658a5176dce8dcbb30b39534a5674c9fb7" exitCode=0
Sep 09 20:31:31.589699 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.589637    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" event={"ID":"29a1a317-2751-4ae6-8711-d0806d145ba9","Type":"ContainerDied","Data":"c4d4406dfe1a3d62ce32050c5c6436658a5176dce8dcbb30b39534a5674c9fb7"}
Sep 09 20:31:31.592377 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.592349    2579 generic.go:356] "Generic (PLEG): container finished" podID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerID="09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3" exitCode=0
Sep 09 20:31:31.592476 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.592419    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" event={"ID":"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3","Type":"ContainerDied","Data":"09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3"}
Sep 09 20:31:31.592476 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.592427    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"
Sep 09 20:31:31.592588 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.592477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d" event={"ID":"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3","Type":"ContainerDied","Data":"2e86ddf803834bc16a85f173f7a5eb9cb8b530d35984f5122b0e2eec7b5a66d1"}
Sep 09 20:31:31.592588 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.592514    2579 scope.go:117] "RemoveContainer" containerID="b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912"
Sep 09 20:31:31.636970 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.636950    2579 scope.go:117] "RemoveContainer" containerID="09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3"
Sep 09 20:31:31.657943 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.657920    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcdhj\" (UniqueName: \"kubernetes.io/projected/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-api-access-fcdhj\") pod \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") "
Sep 09 20:31:31.658033 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.657964    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-cabundle-cert\") pod \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") "
Sep 09 20:31:31.658505 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.658103    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-proxy-tls\") pod \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") "
Sep 09 20:31:31.658505 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.658193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-rbac-proxy-sar-config\") pod \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") "
Sep 09 20:31:31.658505 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.658252    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kserve-provision-location\") pod \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\" (UID: \"cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3\") "
Sep 09 20:31:31.658505 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.658332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" (UID: "cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:31.658505 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.658479    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:31:31.658998 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.658651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" (UID: "cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:31:31.658998 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.658685    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" (UID: "cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:31.660207 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.660181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-api-access-fcdhj" (OuterVolumeSpecName: "kube-api-access-fcdhj") pod "cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" (UID: "cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3"). InnerVolumeSpecName "kube-api-access-fcdhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:31.660793 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.660773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" (UID: "cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:31:31.662795 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.662778    2579 scope.go:117] "RemoveContainer" containerID="c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76"
Sep 09 20:31:31.676834 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.676813    2579 scope.go:117] "RemoveContainer" containerID="b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912"
Sep 09 20:31:31.677104 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:31:31.677074    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912\": container with ID starting with b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912 not found: ID does not exist" containerID="b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912"
Sep 09 20:31:31.677223 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.677102    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912"} err="failed to get container status \"b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912\": rpc error: code = NotFound desc = could not find container \"b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912\": container with ID starting with b8fb0999574e8a7433c217e5db5320837315473724896c30b7cba598b1cca912 not found: ID does not exist"
Sep 09 20:31:31.677223 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.677120    2579 scope.go:117] "RemoveContainer" containerID="09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3"
Sep 09 20:31:31.677424 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:31:31.677408    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3\": container with ID starting with 09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3 not found: ID does not exist" containerID="09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3"
Sep 09 20:31:31.677464 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.677430    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3"} err="failed to get container status \"09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3\": rpc error: code = NotFound desc = could not find container \"09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3\": container with ID starting with 09de257c9c1142294d8a8aca632c1c81c3e865dfb9bdb4bf345dc61187da43e3 not found: ID does not exist"
Sep 09 20:31:31.677464 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.677444    2579 scope.go:117] "RemoveContainer" containerID="c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76"
Sep 09 20:31:31.677664 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:31:31.677649    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76\": container with ID starting with c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76 not found: ID does not exist" containerID="c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76"
Sep 09 20:31:31.677710 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.677677    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76"} err="failed to get container status \"c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76\": rpc error: code = NotFound desc = could not find container \"c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76\": container with ID starting with c24d31e90bcb0282363c018261695e0b7f5b5352b688b1bfaa1dae651801ff76 not found: ID does not exist"
Sep 09 20:31:31.758884 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.758860    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:31:31.758996 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.758889    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:31:31.758996 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.758906    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcdhj\" (UniqueName: \"kubernetes.io/projected/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-kube-api-access-fcdhj\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:31:31.758996 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.758920    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:31:31.912552 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.912519    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"]
Sep 09 20:31:31.915993 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:31.915972    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-38f5d-predictor-9985b5fc8-b7n7d"]
Sep 09 20:31:32.117049 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:32.117018    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" path="/var/lib/kubelet/pods/cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3/volumes"
Sep 09 20:31:32.598057 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:32.597975    2579 generic.go:356] "Generic (PLEG): container finished" podID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerID="d7f62b3ca2c7ca752f90d91d6b1325c783e71c1e4540c345f4089638f81da329" exitCode=0
Sep 09 20:31:32.598479 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:32.598050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" event={"ID":"ce2c60ca-0045-4948-a75a-3aafa3eca2f2","Type":"ContainerDied","Data":"d7f62b3ca2c7ca752f90d91d6b1325c783e71c1e4540c345f4089638f81da329"}
Sep 09 20:31:32.600266 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:32.600241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" event={"ID":"29a1a317-2751-4ae6-8711-d0806d145ba9","Type":"ContainerStarted","Data":"cdee919e4b32260dd08da1967bde5dfac9505231c4f10133c11f42bba680fe8c"}
Sep 09 20:31:32.600391 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:32.600302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" event={"ID":"29a1a317-2751-4ae6-8711-d0806d145ba9","Type":"ContainerStarted","Data":"758971ba4a55acc82a679b9e2710ecfa1a509d728dfb1722128b0c435b2f7571"}
Sep 09 20:31:32.600528 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:32.600508    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:32.631521 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:32.631484    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podStartSLOduration=2.864286198 podStartE2EDuration="6.631470596s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.767184398s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:32.62993248 +0000 UTC m=+1227.006059226" watchObservedRunningTime="2026-09-09 20:31:32.631470596 +0000 UTC m=+1227.007597342"
Sep 09 20:31:33.606680 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:33.606646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" event={"ID":"ce2c60ca-0045-4948-a75a-3aafa3eca2f2","Type":"ContainerStarted","Data":"b92ef7891d6ac3c679759e44fea6f48c8c55a9105843798a7dfce83f2c71d8c8"}
Sep 09 20:31:33.607122 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:33.606705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" event={"ID":"ce2c60ca-0045-4948-a75a-3aafa3eca2f2","Type":"ContainerStarted","Data":"14206c73c2ef61def3f5f4d38456d7994672540f73ca509844037f67a467db67"}
Sep 09 20:31:33.607257 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:33.607234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:33.607328 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:33.607289    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:33.607328 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:33.607322    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:33.608065 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:33.608042    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:31:33.608141 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:33.608103    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:31:33.624484 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:33.624442    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podStartSLOduration=3.615691472 podStartE2EDuration="7.624426873s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.008735401s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:33.62287034 +0000 UTC m=+1227.998997086" watchObservedRunningTime="2026-09-09 20:31:33.624426873 +0000 UTC m=+1228.000553643"
Sep 09 20:31:34.610734 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:34.610700    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:31:34.611146 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:34.610731    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:31:39.614798 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:39.614760    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:31:39.615281 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:39.615089    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:31:39.615281 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:39.615235    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:31:39.615644 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:39.615624    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:31:49.615187 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:49.615152    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:31:49.615576 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:49.615551    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:31:59.615262 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:59.615222    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:31:59.615645 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:31:59.615517    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:32:09.615756 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:09.615720    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:32:09.616207 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:09.615720    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:32:19.615235 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:19.615198    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:32:19.615616 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:19.615466    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:32:29.615210 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:29.615178    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:32:29.615633 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:29.615498    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:32:39.615268 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:39.615226    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:32:39.615716 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:39.615523    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:32:49.615628 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:49.615593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:32:49.616301 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:49.616279    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:32:56.982080 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:56.982049    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"]
Sep 09 20:32:56.982509 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:56.982322    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container" containerID="cri-o://14206c73c2ef61def3f5f4d38456d7994672540f73ca509844037f67a467db67" gracePeriod=30
Sep 09 20:32:56.982509 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:56.982364    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kube-rbac-proxy" containerID="cri-o://b92ef7891d6ac3c679759e44fea6f48c8c55a9105843798a7dfce83f2c71d8c8" gracePeriod=30
Sep 09 20:32:56.989680 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:56.989655    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"]
Sep 09 20:32:56.989945 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:56.989923    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container" containerID="cri-o://758971ba4a55acc82a679b9e2710ecfa1a509d728dfb1722128b0c435b2f7571" gracePeriod=30
Sep 09 20:32:56.990052 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:56.990009    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kube-rbac-proxy" containerID="cri-o://cdee919e4b32260dd08da1967bde5dfac9505231c4f10133c11f42bba680fe8c" gracePeriod=30
Sep 09 20:32:57.008470 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:57.008448    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:57.912070 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:57.912038    2579 generic.go:356] "Generic (PLEG): container finished" podID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerID="b92ef7891d6ac3c679759e44fea6f48c8c55a9105843798a7dfce83f2c71d8c8" exitCode=2
Sep 09 20:32:57.912278 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:57.912110    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" event={"ID":"ce2c60ca-0045-4948-a75a-3aafa3eca2f2","Type":"ContainerDied","Data":"b92ef7891d6ac3c679759e44fea6f48c8c55a9105843798a7dfce83f2c71d8c8"}
Sep 09 20:32:57.914074 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:57.914054    2579 generic.go:356] "Generic (PLEG): container finished" podID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerID="cdee919e4b32260dd08da1967bde5dfac9505231c4f10133c11f42bba680fe8c" exitCode=2
Sep 09 20:32:57.914208 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:57.914120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" event={"ID":"29a1a317-2751-4ae6-8711-d0806d145ba9","Type":"ContainerDied","Data":"cdee919e4b32260dd08da1967bde5dfac9505231c4f10133c11f42bba680fe8c"}
Sep 09 20:32:58.605722 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:58.605692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:32:59.611502 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:59.611463    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 09 20:32:59.611904 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:59.611463    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 09 20:32:59.615694 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:59.615672    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 09 20:32:59.615759 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:32:59.615673    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 09 20:33:01.929978 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:01.929948    2579 generic.go:356] "Generic (PLEG): container finished" podID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerID="14206c73c2ef61def3f5f4d38456d7994672540f73ca509844037f67a467db67" exitCode=0
Sep 09 20:33:01.930358 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:01.930022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" event={"ID":"ce2c60ca-0045-4948-a75a-3aafa3eca2f2","Type":"ContainerDied","Data":"14206c73c2ef61def3f5f4d38456d7994672540f73ca509844037f67a467db67"}
Sep 09 20:33:01.932331 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:01.932309    2579 generic.go:356] "Generic (PLEG): container finished" podID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerID="758971ba4a55acc82a679b9e2710ecfa1a509d728dfb1722128b0c435b2f7571" exitCode=0
Sep 09 20:33:01.932454 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:01.932381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" event={"ID":"29a1a317-2751-4ae6-8711-d0806d145ba9","Type":"ContainerDied","Data":"758971ba4a55acc82a679b9e2710ecfa1a509d728dfb1722128b0c435b2f7571"}
Sep 09 20:33:01.948726 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:01.948704    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:33:01.951532 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:01.951514    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:33:02.109568 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.109482    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:02.123810 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.123787    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a1a317-2751-4ae6-8711-d0806d145ba9-proxy-tls\") pod \"29a1a317-2751-4ae6-8711-d0806d145ba9\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") "
Sep 09 20:33:02.123969 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.123822    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a1a317-2751-4ae6-8711-d0806d145ba9-kserve-provision-location\") pod \"29a1a317-2751-4ae6-8711-d0806d145ba9\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") "
Sep 09 20:33:02.123969 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.123844    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-proxy-tls\") pod \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") "
Sep 09 20:33:02.123969 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.123878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-cabundle-cert\") pod \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") "
Sep 09 20:33:02.123969 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.123902    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kserve-provision-location\") pod \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") "
Sep 09 20:33:02.124234 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.123982    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-cabundle-cert\") pod \"29a1a317-2751-4ae6-8711-d0806d145ba9\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") "
Sep 09 20:33:02.124234 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.124013    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6qfb\" (UniqueName: \"kubernetes.io/projected/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-api-access-b6qfb\") pod \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") "
Sep 09 20:33:02.124234 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.124049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-rbac-proxy-sar-config\") pod \"29a1a317-2751-4ae6-8711-d0806d145ba9\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") "
Sep 09 20:33:02.124234 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.124078    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-rbac-proxy-sar-config\") pod \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\" (UID: \"ce2c60ca-0045-4948-a75a-3aafa3eca2f2\") "
Sep 09 20:33:02.124234 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.124103    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64vh4\" (UniqueName: \"kubernetes.io/projected/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-api-access-64vh4\") pod \"29a1a317-2751-4ae6-8711-d0806d145ba9\" (UID: \"29a1a317-2751-4ae6-8711-d0806d145ba9\") "
Sep 09 20:33:02.124234 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.124186    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a1a317-2751-4ae6-8711-d0806d145ba9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29a1a317-2751-4ae6-8711-d0806d145ba9" (UID: "29a1a317-2751-4ae6-8711-d0806d145ba9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:02.124563 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.124358    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce2c60ca-0045-4948-a75a-3aafa3eca2f2" (UID: "ce2c60ca-0045-4948-a75a-3aafa3eca2f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:33:02.124563 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.124391    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29a1a317-2751-4ae6-8711-d0806d145ba9-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:33:02.124563 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.124387    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce2c60ca-0045-4948-a75a-3aafa3eca2f2" (UID: "ce2c60ca-0045-4948-a75a-3aafa3eca2f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:02.124747 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.124714    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce2c60ca-0045-4948-a75a-3aafa3eca2f2" (UID: "ce2c60ca-0045-4948-a75a-3aafa3eca2f2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:33:02.124831 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.124740    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29a1a317-2751-4ae6-8711-d0806d145ba9" (UID: "29a1a317-2751-4ae6-8711-d0806d145ba9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:33:02.124831 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.124756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29a1a317-2751-4ae6-8711-d0806d145ba9" (UID: "29a1a317-2751-4ae6-8711-d0806d145ba9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:33:02.126374 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.126349    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-api-access-64vh4" (OuterVolumeSpecName: "kube-api-access-64vh4") pod "29a1a317-2751-4ae6-8711-d0806d145ba9" (UID: "29a1a317-2751-4ae6-8711-d0806d145ba9"). InnerVolumeSpecName "kube-api-access-64vh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:33:02.126613 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.126587    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-api-access-b6qfb" (OuterVolumeSpecName: "kube-api-access-b6qfb") pod "ce2c60ca-0045-4948-a75a-3aafa3eca2f2" (UID: "ce2c60ca-0045-4948-a75a-3aafa3eca2f2"). InnerVolumeSpecName "kube-api-access-b6qfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:33:02.126701 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.126657    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce2c60ca-0045-4948-a75a-3aafa3eca2f2" (UID: "ce2c60ca-0045-4948-a75a-3aafa3eca2f2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:33:02.126744 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.126714    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a1a317-2751-4ae6-8711-d0806d145ba9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29a1a317-2751-4ae6-8711-d0806d145ba9" (UID: "29a1a317-2751-4ae6-8711-d0806d145ba9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:33:02.191576 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.191538    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:02.224933 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.224897    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:33:02.224933 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.224926    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:33:02.224933 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.224939    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:33:02.225224 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.224952    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:33:02.225224 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.224965    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6qfb\" (UniqueName: \"kubernetes.io/projected/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-api-access-b6qfb\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:33:02.225224 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.224976    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:33:02.225224 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.224989    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce2c60ca-0045-4948-a75a-3aafa3eca2f2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:33:02.225224 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225000    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64vh4\" (UniqueName: \"kubernetes.io/projected/29a1a317-2751-4ae6-8711-d0806d145ba9-kube-api-access-64vh4\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:33:02.225224 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225010    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29a1a317-2751-4ae6-8711-d0806d145ba9-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:33:02.225224 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225175    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"]
Sep 09 20:33:02.225431 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225413    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="storage-initializer"
Sep 09 20:33:02.225431 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225423    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="storage-initializer"
Sep 09 20:33:02.225492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225437    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="storage-initializer"
Sep 09 20:33:02.225492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225443    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="storage-initializer"
Sep 09 20:33:02.225492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225450    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container"
Sep 09 20:33:02.225492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225455    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container"
Sep 09 20:33:02.225492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225460    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container"
Sep 09 20:33:02.225492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225465    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container"
Sep 09 20:33:02.225492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225472    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kube-rbac-proxy"
Sep 09 20:33:02.225492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225477    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kube-rbac-proxy"
Sep 09 20:33:02.225492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225485    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="storage-initializer"
Sep 09 20:33:02.225492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225490    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="storage-initializer"
Sep 09 20:33:02.225492 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225496    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kube-rbac-proxy"
Sep 09 20:33:02.225804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225500    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kube-rbac-proxy"
Sep 09 20:33:02.225804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225508    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container"
Sep 09 20:33:02.225804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225513    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container"
Sep 09 20:33:02.225804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225519    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kube-rbac-proxy"
Sep 09 20:33:02.225804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225524    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kube-rbac-proxy"
Sep 09 20:33:02.225804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225560    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kserve-container"
Sep 09 20:33:02.225804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225566    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc8c3144-a6a7-4a98-b9c3-82467f0a4cb3" containerName="kube-rbac-proxy"
Sep 09 20:33:02.225804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225571    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kube-rbac-proxy"
Sep 09 20:33:02.225804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225577    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kube-rbac-proxy"
Sep 09 20:33:02.225804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225584    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" containerName="kserve-container"
Sep 09 20:33:02.225804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.225589    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" containerName="kserve-container"
Sep 09 20:33:02.229870 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.229853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.232003 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.231980    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 20:33:02.232112 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.232042    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 20:33:02.238452 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.238429    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"]
Sep 09 20:33:02.262591 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.262554    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:02.426913 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.426880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.426913 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.426923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e121491-be4e-48a2-84d4-3841db5336a6-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.427198 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.426962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.427198 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.427012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e121491-be4e-48a2-84d4-3841db5336a6-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.427198 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.427040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw8k8\" (UniqueName: \"kubernetes.io/projected/6e121491-be4e-48a2-84d4-3841db5336a6-kube-api-access-hw8k8\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.527535 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.527495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e121491-be4e-48a2-84d4-3841db5336a6-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.527535 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.527538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw8k8\" (UniqueName: \"kubernetes.io/projected/6e121491-be4e-48a2-84d4-3841db5336a6-kube-api-access-hw8k8\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.527764 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.527582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.527764 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.527600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e121491-be4e-48a2-84d4-3841db5336a6-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.527764 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.527621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.527764 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:33:02.527752    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 20:33:02.527995 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:33:02.527819    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e121491-be4e-48a2-84d4-3841db5336a6-proxy-tls podName:6e121491-be4e-48a2-84d4-3841db5336a6 nodeName:}" failed. No retries permitted until 2026-09-09 20:33:03.027797242 +0000 UTC m=+1317.403923973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6e121491-be4e-48a2-84d4-3841db5336a6-proxy-tls") pod "isvc-canary-promote-predictor-648d7cfc8f-w95pq" (UID: "6e121491-be4e-48a2-84d4-3841db5336a6") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 09 20:33:02.527995 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.527897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e121491-be4e-48a2-84d4-3841db5336a6-kserve-provision-location\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.528275 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.528255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.528391 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.528334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-cabundle-cert\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.536693 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.536671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw8k8\" (UniqueName: \"kubernetes.io/projected/6e121491-be4e-48a2-84d4-3841db5336a6-kube-api-access-hw8k8\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:02.587336 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.587311    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:02.937480 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.937451    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"
Sep 09 20:33:02.937899 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.937447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74" event={"ID":"ce2c60ca-0045-4948-a75a-3aafa3eca2f2","Type":"ContainerDied","Data":"a262bb2f0043e55409d1492dc8010c54f66d47be4f96368d8e9b8c3d9e5c3d72"}
Sep 09 20:33:02.937899 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.937616    2579 scope.go:117] "RemoveContainer" containerID="b92ef7891d6ac3c679759e44fea6f48c8c55a9105843798a7dfce83f2c71d8c8"
Sep 09 20:33:02.939917 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.939891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb" event={"ID":"29a1a317-2751-4ae6-8711-d0806d145ba9","Type":"ContainerDied","Data":"1b561749e441a29da77a61fd629da5fc575a10a534ad66c6d64be2861867bf98"}
Sep 09 20:33:02.940057 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.939901    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"
Sep 09 20:33:02.945942 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.945927    2579 scope.go:117] "RemoveContainer" containerID="14206c73c2ef61def3f5f4d38456d7994672540f73ca509844037f67a467db67"
Sep 09 20:33:02.953985 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.953968    2579 scope.go:117] "RemoveContainer" containerID="d7f62b3ca2c7ca752f90d91d6b1325c783e71c1e4540c345f4089638f81da329"
Sep 09 20:33:02.960796 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.960779    2579 scope.go:117] "RemoveContainer" containerID="cdee919e4b32260dd08da1967bde5dfac9505231c4f10133c11f42bba680fe8c"
Sep 09 20:33:02.963370 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.963349    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"]
Sep 09 20:33:02.967332 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.967313    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-559cdfb684-42k74"]
Sep 09 20:33:02.968222 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.968204    2579 scope.go:117] "RemoveContainer" containerID="758971ba4a55acc82a679b9e2710ecfa1a509d728dfb1722128b0c435b2f7571"
Sep 09 20:33:02.974949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.974932    2579 scope.go:117] "RemoveContainer" containerID="c4d4406dfe1a3d62ce32050c5c6436658a5176dce8dcbb30b39534a5674c9fb7"
Sep 09 20:33:02.977172 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.977151    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"]
Sep 09 20:33:02.979354 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:02.979332    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5f7b4999bf-jg2xb"]
Sep 09 20:33:03.031090 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:03.031046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e121491-be4e-48a2-84d4-3841db5336a6-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:03.033550 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:03.033490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e121491-be4e-48a2-84d4-3841db5336a6-proxy-tls\") pod \"isvc-canary-promote-predictor-648d7cfc8f-w95pq\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:03.140402 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:03.140355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:03.415642 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:03.415463    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"]
Sep 09 20:33:03.418123 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:33:03.418097    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e121491_be4e_48a2_84d4_3841db5336a6.slice/crio-cb86252783092b0b9aeab55d5dd0d1e2b3844089c2f1433c47b3ddb5826d9379 WatchSource:0}: Error finding container cb86252783092b0b9aeab55d5dd0d1e2b3844089c2f1433c47b3ddb5826d9379: Status 404 returned error can't find the container with id cb86252783092b0b9aeab55d5dd0d1e2b3844089c2f1433c47b3ddb5826d9379
Sep 09 20:33:03.947103 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:03.947063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" event={"ID":"6e121491-be4e-48a2-84d4-3841db5336a6","Type":"ContainerStarted","Data":"ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3"}
Sep 09 20:33:03.947462 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:03.947136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" event={"ID":"6e121491-be4e-48a2-84d4-3841db5336a6","Type":"ContainerStarted","Data":"cb86252783092b0b9aeab55d5dd0d1e2b3844089c2f1433c47b3ddb5826d9379"}
Sep 09 20:33:04.117372 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:04.117343    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a1a317-2751-4ae6-8711-d0806d145ba9" path="/var/lib/kubelet/pods/29a1a317-2751-4ae6-8711-d0806d145ba9/volumes"
Sep 09 20:33:04.117863 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:04.117850    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce2c60ca-0045-4948-a75a-3aafa3eca2f2" path="/var/lib/kubelet/pods/ce2c60ca-0045-4948-a75a-3aafa3eca2f2/volumes"
Sep 09 20:33:06.112673 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:06.112633    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:07.961958 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:07.961924    2579 generic.go:356] "Generic (PLEG): container finished" podID="6e121491-be4e-48a2-84d4-3841db5336a6" containerID="ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3" exitCode=0
Sep 09 20:33:07.962398 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:07.961996    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" event={"ID":"6e121491-be4e-48a2-84d4-3841db5336a6","Type":"ContainerDied","Data":"ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3"}
Sep 09 20:33:08.967766 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:08.967732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" event={"ID":"6e121491-be4e-48a2-84d4-3841db5336a6","Type":"ContainerStarted","Data":"477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044"}
Sep 09 20:33:08.968183 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:08.967791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" event={"ID":"6e121491-be4e-48a2-84d4-3841db5336a6","Type":"ContainerStarted","Data":"7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255"}
Sep 09 20:33:08.968183 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:08.967992    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:08.987841 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:08.987794    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podStartSLOduration=3.169303042 podStartE2EDuration="6.987777192s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81847415s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:08.986028596 +0000 UTC m=+1323.362155342" watchObservedRunningTime="2026-09-09 20:33:08.987777192 +0000 UTC m=+1323.363903990"
Sep 09 20:33:09.971676 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:09.971631    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:09.972965 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:09.972940    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:33:10.975644 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:10.975606    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:33:15.980027 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:15.979999    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:33:15.980566 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:15.980541    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:33:18.284837 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:18.284802    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:33:25.980743 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:25.980695    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:33:35.981008 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:35.980964    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:33:45.980691 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:45.980646    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:33:55.981319 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:33:55.981279    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:34:05.980561 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:05.980513    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:34:15.981359 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:15.981320    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:34:25.113463 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:25.113431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:34:32.258640 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.258599    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"]
Sep 09 20:34:32.259152 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.258931    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container" containerID="cri-o://7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255" gracePeriod=30
Sep 09 20:34:32.259152 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.258939    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kube-rbac-proxy" containerID="cri-o://477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044" gracePeriod=30
Sep 09 20:34:32.281281 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.281245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:34:32.321194 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.321113    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"]
Sep 09 20:34:32.324702 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.324675    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.326898 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.326875    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 20:34:32.333987 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.333966    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"]
Sep 09 20:34:32.467187 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.467152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.467336 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.467197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.467336 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.467218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7jmc\" (UniqueName: \"kubernetes.io/projected/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-api-access-v7jmc\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.467336 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.467239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14d234c7-48ae-4a86-bc22-c2e260bddcbd-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.467336 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.467254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.568623 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.568548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14d234c7-48ae-4a86-bc22-c2e260bddcbd-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.568623 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.568581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.568781 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.568634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.568820 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.568791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.568856 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.568827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7jmc\" (UniqueName: \"kubernetes.io/projected/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-api-access-v7jmc\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.569062 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.569034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.569409 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.569387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.569502 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.569403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.571078 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.571061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14d234c7-48ae-4a86-bc22-c2e260bddcbd-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.577428 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.577398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7jmc\" (UniqueName: \"kubernetes.io/projected/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-api-access-v7jmc\") pod \"isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.636256 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.636233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:32.908143 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:32.908108    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"]
Sep 09 20:34:32.910314 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:34:32.910285    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d234c7_48ae_4a86_bc22_c2e260bddcbd.slice/crio-d4cd02f75d2540ad22f48b64f9ff70e6d4cc51c02abb8ee505c86f494d2dd92f WatchSource:0}: Error finding container d4cd02f75d2540ad22f48b64f9ff70e6d4cc51c02abb8ee505c86f494d2dd92f: Status 404 returned error can't find the container with id d4cd02f75d2540ad22f48b64f9ff70e6d4cc51c02abb8ee505c86f494d2dd92f
Sep 09 20:34:33.252933 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:33.252836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" event={"ID":"14d234c7-48ae-4a86-bc22-c2e260bddcbd","Type":"ContainerStarted","Data":"ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db"}
Sep 09 20:34:33.252933 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:33.252908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" event={"ID":"14d234c7-48ae-4a86-bc22-c2e260bddcbd","Type":"ContainerStarted","Data":"d4cd02f75d2540ad22f48b64f9ff70e6d4cc51c02abb8ee505c86f494d2dd92f"}
Sep 09 20:34:33.254965 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:33.254943    2579 generic.go:356] "Generic (PLEG): container finished" podID="6e121491-be4e-48a2-84d4-3841db5336a6" containerID="477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044" exitCode=2
Sep 09 20:34:33.255088 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:33.254977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" event={"ID":"6e121491-be4e-48a2-84d4-3841db5336a6","Type":"ContainerDied","Data":"477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044"}
Sep 09 20:34:35.112751 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:35.112705    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 09 20:34:35.976617 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:35.976573    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 09 20:34:36.895511 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:36.895491    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:34:37.001053 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.000973    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw8k8\" (UniqueName: \"kubernetes.io/projected/6e121491-be4e-48a2-84d4-3841db5336a6-kube-api-access-hw8k8\") pod \"6e121491-be4e-48a2-84d4-3841db5336a6\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") "
Sep 09 20:34:37.001217 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.001060    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e121491-be4e-48a2-84d4-3841db5336a6-kserve-provision-location\") pod \"6e121491-be4e-48a2-84d4-3841db5336a6\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") "
Sep 09 20:34:37.001217 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.001177    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-kube-rbac-proxy-sar-config\") pod \"6e121491-be4e-48a2-84d4-3841db5336a6\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") "
Sep 09 20:34:37.001320 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.001226    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e121491-be4e-48a2-84d4-3841db5336a6-proxy-tls\") pod \"6e121491-be4e-48a2-84d4-3841db5336a6\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") "
Sep 09 20:34:37.001320 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.001256    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-cabundle-cert\") pod \"6e121491-be4e-48a2-84d4-3841db5336a6\" (UID: \"6e121491-be4e-48a2-84d4-3841db5336a6\") "
Sep 09 20:34:37.001428 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.001323    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e121491-be4e-48a2-84d4-3841db5336a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e121491-be4e-48a2-84d4-3841db5336a6" (UID: "6e121491-be4e-48a2-84d4-3841db5336a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:34:37.001495 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.001477    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e121491-be4e-48a2-84d4-3841db5336a6-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:34:37.001563 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.001545    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e121491-be4e-48a2-84d4-3841db5336a6" (UID: "6e121491-be4e-48a2-84d4-3841db5336a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:37.001663 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.001644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e121491-be4e-48a2-84d4-3841db5336a6" (UID: "6e121491-be4e-48a2-84d4-3841db5336a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:37.003245 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.003226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e121491-be4e-48a2-84d4-3841db5336a6-kube-api-access-hw8k8" (OuterVolumeSpecName: "kube-api-access-hw8k8") pod "6e121491-be4e-48a2-84d4-3841db5336a6" (UID: "6e121491-be4e-48a2-84d4-3841db5336a6"). InnerVolumeSpecName "kube-api-access-hw8k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:37.003323 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.003303    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e121491-be4e-48a2-84d4-3841db5336a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e121491-be4e-48a2-84d4-3841db5336a6" (UID: "6e121491-be4e-48a2-84d4-3841db5336a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:37.102170 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.102118    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e121491-be4e-48a2-84d4-3841db5336a6-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:34:37.102170 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.102164    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:34:37.102170 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.102174    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw8k8\" (UniqueName: \"kubernetes.io/projected/6e121491-be4e-48a2-84d4-3841db5336a6-kube-api-access-hw8k8\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:34:37.102387 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.102183    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e121491-be4e-48a2-84d4-3841db5336a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:34:37.269899 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.269811    2579 generic.go:356] "Generic (PLEG): container finished" podID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerID="ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db" exitCode=0
Sep 09 20:34:37.270067 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.269891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" event={"ID":"14d234c7-48ae-4a86-bc22-c2e260bddcbd","Type":"ContainerDied","Data":"ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db"}
Sep 09 20:34:37.271845 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.271826    2579 generic.go:356] "Generic (PLEG): container finished" podID="6e121491-be4e-48a2-84d4-3841db5336a6" containerID="7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255" exitCode=0
Sep 09 20:34:37.271968 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.271847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" event={"ID":"6e121491-be4e-48a2-84d4-3841db5336a6","Type":"ContainerDied","Data":"7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255"}
Sep 09 20:34:37.271968 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.271885    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"
Sep 09 20:34:37.271968 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.271895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq" event={"ID":"6e121491-be4e-48a2-84d4-3841db5336a6","Type":"ContainerDied","Data":"cb86252783092b0b9aeab55d5dd0d1e2b3844089c2f1433c47b3ddb5826d9379"}
Sep 09 20:34:37.271968 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.271922    2579 scope.go:117] "RemoveContainer" containerID="477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044"
Sep 09 20:34:37.281899 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.281882    2579 scope.go:117] "RemoveContainer" containerID="7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255"
Sep 09 20:34:37.291040 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.290925    2579 scope.go:117] "RemoveContainer" containerID="ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3"
Sep 09 20:34:37.299960 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.299938    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"]
Sep 09 20:34:37.304409 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.304391    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-648d7cfc8f-w95pq"]
Sep 09 20:34:37.307901 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.307880    2579 scope.go:117] "RemoveContainer" containerID="477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044"
Sep 09 20:34:37.308224 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:34:37.308195    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044\": container with ID starting with 477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044 not found: ID does not exist" containerID="477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044"
Sep 09 20:34:37.308312 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.308227    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044"} err="failed to get container status \"477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044\": rpc error: code = NotFound desc = could not find container \"477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044\": container with ID starting with 477ec637d18b3fc62134a25e352c7c43e1eb2053da00c25f3a723a65909e7044 not found: ID does not exist"
Sep 09 20:34:37.308312 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.308248    2579 scope.go:117] "RemoveContainer" containerID="7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255"
Sep 09 20:34:37.308573 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:34:37.308544    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255\": container with ID starting with 7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255 not found: ID does not exist" containerID="7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255"
Sep 09 20:34:37.308670 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.308581    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255"} err="failed to get container status \"7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255\": rpc error: code = NotFound desc = could not find container \"7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255\": container with ID starting with 7e11e22d3190bc0dfd7ac16b88fa2d5d9fb2a6ca2539c4eda7bf823e73bc0255 not found: ID does not exist"
Sep 09 20:34:37.308670 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.308603    2579 scope.go:117] "RemoveContainer" containerID="ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3"
Sep 09 20:34:37.308922 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:34:37.308907    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3\": container with ID starting with ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3 not found: ID does not exist" containerID="ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3"
Sep 09 20:34:37.308967 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:37.308927    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3"} err="failed to get container status \"ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3\": rpc error: code = NotFound desc = could not find container \"ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3\": container with ID starting with ddd6fe78eee57642ca4406ce51de1812131ec860b5377b1a4a7ff4a6d8745ec3 not found: ID does not exist"
Sep 09 20:34:38.116407 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:38.116374    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" path="/var/lib/kubelet/pods/6e121491-be4e-48a2-84d4-3841db5336a6/volumes"
Sep 09 20:34:38.278392 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:38.278357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" event={"ID":"14d234c7-48ae-4a86-bc22-c2e260bddcbd","Type":"ContainerStarted","Data":"dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc"}
Sep 09 20:34:38.278565 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:38.278416    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" event={"ID":"14d234c7-48ae-4a86-bc22-c2e260bddcbd","Type":"ContainerStarted","Data":"67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912"}
Sep 09 20:34:38.278721 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:38.278698    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:38.278804 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:38.278756    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:38.280019 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:38.279989    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:34:38.296281 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:38.296237    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podStartSLOduration=2.452889358 podStartE2EDuration="6.296224658s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8433353s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:38.29531375 +0000 UTC m=+1412.671440508" watchObservedRunningTime="2026-09-09 20:34:38.296224658 +0000 UTC m=+1412.672351402"
Sep 09 20:34:39.282007 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:39.281969    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:34:44.287255 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:44.287225    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:34:44.287907 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:44.287876    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:34:54.288400 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:34:54.288353    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:35:04.288459 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:35:04.288419    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:35:14.288803 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:35:14.288760    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:35:24.288113 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:35:24.288077    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:35:28.828331 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:35:28.828263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:32.050498 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:35:32.050460    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:32.108141 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:35:32.108103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:34.288552 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:35:34.288516    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:35:44.287890 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:35:44.287848    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:35:50.401541 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:35:50.401507    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:35:54.289581 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:35:54.289555    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:36:02.486349 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:02.486314    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"]
Sep 09 20:36:02.486760 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:02.486553    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container" containerID="cri-o://67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912" gracePeriod=30
Sep 09 20:36:02.486760 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:02.486604    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kube-rbac-proxy" containerID="cri-o://dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc" gracePeriod=30
Sep 09 20:36:02.712172 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:02.712111    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:03.575246 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:03.575212    2579 generic.go:356] "Generic (PLEG): container finished" podID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerID="dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc" exitCode=2
Sep 09 20:36:03.575634 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:03.575259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" event={"ID":"14d234c7-48ae-4a86-bc22-c2e260bddcbd","Type":"ContainerDied","Data":"dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc"}
Sep 09 20:36:04.282938 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:04.282898    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 09 20:36:04.288289 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:04.288261    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 09 20:36:07.325357 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.325333    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:36:07.463738 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.463650    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kserve-provision-location\") pod \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") "
Sep 09 20:36:07.463738 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.463703    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-cabundle-cert\") pod \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") "
Sep 09 20:36:07.463950 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.463751    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14d234c7-48ae-4a86-bc22-c2e260bddcbd-proxy-tls\") pod \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") "
Sep 09 20:36:07.463950 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.463776    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7jmc\" (UniqueName: \"kubernetes.io/projected/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-api-access-v7jmc\") pod \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") "
Sep 09 20:36:07.463950 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.463815    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-rbac-proxy-sar-config\") pod \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\" (UID: \"14d234c7-48ae-4a86-bc22-c2e260bddcbd\") "
Sep 09 20:36:07.464064 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.464006    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14d234c7-48ae-4a86-bc22-c2e260bddcbd" (UID: "14d234c7-48ae-4a86-bc22-c2e260bddcbd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:36:07.464197 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.464164    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14d234c7-48ae-4a86-bc22-c2e260bddcbd" (UID: "14d234c7-48ae-4a86-bc22-c2e260bddcbd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:07.464309 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.464245    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "14d234c7-48ae-4a86-bc22-c2e260bddcbd" (UID: "14d234c7-48ae-4a86-bc22-c2e260bddcbd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:36:07.466018 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.466000    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d234c7-48ae-4a86-bc22-c2e260bddcbd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14d234c7-48ae-4a86-bc22-c2e260bddcbd" (UID: "14d234c7-48ae-4a86-bc22-c2e260bddcbd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:07.466094 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.466074    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-api-access-v7jmc" (OuterVolumeSpecName: "kube-api-access-v7jmc") pod "14d234c7-48ae-4a86-bc22-c2e260bddcbd" (UID: "14d234c7-48ae-4a86-bc22-c2e260bddcbd"). InnerVolumeSpecName "kube-api-access-v7jmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:07.564769 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.564736    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kserve-provision-location\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:36:07.564769 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.564762    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-cabundle-cert\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:36:07.564769 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.564772    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14d234c7-48ae-4a86-bc22-c2e260bddcbd-proxy-tls\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:36:07.565004 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.564782    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7jmc\" (UniqueName: \"kubernetes.io/projected/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-api-access-v7jmc\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:36:07.565004 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.564790    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14d234c7-48ae-4a86-bc22-c2e260bddcbd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-109.ec2.internal\" DevicePath \"\""
Sep 09 20:36:07.590506 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.590473    2579 generic.go:356] "Generic (PLEG): container finished" podID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerID="67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912" exitCode=0
Sep 09 20:36:07.590639 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.590538    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"
Sep 09 20:36:07.590639 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.590541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" event={"ID":"14d234c7-48ae-4a86-bc22-c2e260bddcbd","Type":"ContainerDied","Data":"67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912"}
Sep 09 20:36:07.590711 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.590673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr" event={"ID":"14d234c7-48ae-4a86-bc22-c2e260bddcbd","Type":"ContainerDied","Data":"d4cd02f75d2540ad22f48b64f9ff70e6d4cc51c02abb8ee505c86f494d2dd92f"}
Sep 09 20:36:07.590711 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.590706    2579 scope.go:117] "RemoveContainer" containerID="dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc"
Sep 09 20:36:07.598670 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.598647    2579 scope.go:117] "RemoveContainer" containerID="67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912"
Sep 09 20:36:07.605510 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.605491    2579 scope.go:117] "RemoveContainer" containerID="ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db"
Sep 09 20:36:07.611229 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.611209    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"]
Sep 09 20:36:07.612841 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.612830    2579 scope.go:117] "RemoveContainer" containerID="dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc"
Sep 09 20:36:07.613089 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:36:07.613074    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc\": container with ID starting with dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc not found: ID does not exist" containerID="dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc"
Sep 09 20:36:07.613153 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.613098    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc"} err="failed to get container status \"dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc\": rpc error: code = NotFound desc = could not find container \"dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc\": container with ID starting with dfa377cb5ab17312e132471a284a62531813ad90c2f503e92d2b0ea10d61f6cc not found: ID does not exist"
Sep 09 20:36:07.613153 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.613147    2579 scope.go:117] "RemoveContainer" containerID="67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912"
Sep 09 20:36:07.613365 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:36:07.613347    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912\": container with ID starting with 67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912 not found: ID does not exist" containerID="67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912"
Sep 09 20:36:07.613437 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.613375    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912"} err="failed to get container status \"67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912\": rpc error: code = NotFound desc = could not find container \"67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912\": container with ID starting with 67b7714569a194e237f8ab9061b11b9c373fde9b28cb00014d073a4ad9287912 not found: ID does not exist"
Sep 09 20:36:07.613437 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.613398    2579 scope.go:117] "RemoveContainer" containerID="ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db"
Sep 09 20:36:07.613624 ip-10-0-132-109 kubenswrapper[2579]: E0909 20:36:07.613606    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db\": container with ID starting with ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db not found: ID does not exist" containerID="ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db"
Sep 09 20:36:07.613662 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.613631    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db"} err="failed to get container status \"ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db\": rpc error: code = NotFound desc = could not find container \"ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db\": container with ID starting with ed25cf12ec4ad6b4a3c99ecfe3b330f3a9499e718a8505133cb1b743191486db not found: ID does not exist"
Sep 09 20:36:07.616935 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:07.616916    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c8d94dbf4-gm4cr"]
Sep 09 20:36:08.116156 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:08.116096    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" path="/var/lib/kubelet/pods/14d234c7-48ae-4a86-bc22-c2e260bddcbd/volumes"
Sep 09 20:36:16.595430 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:16.595394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:16.900031 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:16.899996    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:25.782544 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:25.782508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:36.636462 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:36.636426    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:36:37.062964 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:36:37.062876    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:11.990428 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:11.990399    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:13.567302 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:13.567269    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:43.862872 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:43.862840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:44.003955 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:44.003920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:44.104151 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:44.104104    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:46.797044 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:46.797014    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5l4sk_a213458e-bfea-4d69-aa91-e3a039bfa810/global-pull-secret-syncer/0.log"
Sep 09 20:38:46.901479 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:46.901454    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log"
Sep 09 20:38:46.975246 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:46.975220    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-109.ec2.internal_a72042445d935c31fd1e562dad373422/haproxy/0.log"
Sep 09 20:38:49.388441 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:49.388409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:49.489621 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:49.489594    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4lspj_d3e83679-e37e-4044-8934-54de8aed342a/konnectivity-agent/0.log" limit=1024
Sep 09 20:38:51.392892 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:51.392866    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5b5q_386ad5c3-303f-4a5d-9111-34f6dc9865d5/node-exporter/0.log"
Sep 09 20:38:51.418459 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:51.418433    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5b5q_386ad5c3-303f-4a5d-9111-34f6dc9865d5/kube-rbac-proxy/0.log"
Sep 09 20:38:51.438718 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:51.438694    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f5b5q_386ad5c3-303f-4a5d-9111-34f6dc9865d5/init-textfile/0.log"
Sep 09 20:38:51.804301 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:51.804214    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-wssxb_68683f39-29de-42b4-b847-5e87eec34ca0/prometheus-operator/0.log"
Sep 09 20:38:51.821411 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:51.821391    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-wssxb_68683f39-29de-42b4-b847-5e87eec34ca0/kube-rbac-proxy/0.log"
Sep 09 20:38:55.842517 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842479    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"]
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842714    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kube-rbac-proxy"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842725    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kube-rbac-proxy"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842741    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kube-rbac-proxy"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842747    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kube-rbac-proxy"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842755    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842760    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842767    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="storage-initializer"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842772    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="storage-initializer"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842777    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="storage-initializer"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842782    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="storage-initializer"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842789    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842794    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842827    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kserve-container"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842834    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kserve-container"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842839    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e121491-be4e-48a2-84d4-3841db5336a6" containerName="kube-rbac-proxy"
Sep 09 20:38:55.842916 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.842846    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d234c7-48ae-4a86-bc22-c2e260bddcbd" containerName="kube-rbac-proxy"
Sep 09 20:38:55.845648 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.845627    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.847879 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.847856    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6t7hr\"/\"openshift-service-ca.crt\""
Sep 09 20:38:55.849034 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.849019    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6t7hr\"/\"kube-root-ca.crt\""
Sep 09 20:38:55.849034 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.849031    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6t7hr\"/\"default-dockercfg-lgrms\""
Sep 09 20:38:55.854568 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.854546    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"]
Sep 09 20:38:55.855871 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.855854    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k7j2s_58cb6577-3053-40db-a53e-fff4d4a4a172/dns/0.log"
Sep 09 20:38:55.882949 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.882927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-lib-modules\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.883060 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.882965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-podres\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.883060 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.882985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-proc\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.883060 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.883005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-sys\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.883197 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.883066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2nm4\" (UniqueName: \"kubernetes.io/projected/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-kube-api-access-f2nm4\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.888826 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.888809    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k7j2s_58cb6577-3053-40db-a53e-fff4d4a4a172/kube-rbac-proxy/0.log"
Sep 09 20:38:55.975524 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.975499    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f5dz5_6bb0d4fd-0dd8-43ef-acbb-4e8825c6ff16/dns-node-resolver/0.log"
Sep 09 20:38:55.983884 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.983858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-lib-modules\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.983984 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.983887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-podres\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.983984 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.983908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-proc\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.983984 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.983927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-sys\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.983984 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.983950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2nm4\" (UniqueName: \"kubernetes.io/projected/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-kube-api-access-f2nm4\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.984167 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.984018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-sys\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.984167 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.984028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-lib-modules\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.984167 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.984032    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-podres\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.988538 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.984007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-proc\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:55.993968 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:55.993948    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2nm4\" (UniqueName: \"kubernetes.io/projected/0bc5ac3f-1d1b-48ea-835a-058b59debfbd-kube-api-access-f2nm4\") pod \"perf-node-gather-daemonset-nljzg\" (UID: \"0bc5ac3f-1d1b-48ea-835a-058b59debfbd\") " pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:56.154862 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:56.154828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:56.420596 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:56.420440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"]
Sep 09 20:38:56.423483 ip-10-0-132-109 kubenswrapper[2579]: W0909 20:38:56.423458    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0bc5ac3f_1d1b_48ea_835a_058b59debfbd.slice/crio-ffa19728f66741c9edc26705ac99a0c92efc19556d1e71c4fb484406b4295a51 WatchSource:0}: Error finding container ffa19728f66741c9edc26705ac99a0c92efc19556d1e71c4fb484406b4295a51: Status 404 returned error can't find the container with id ffa19728f66741c9edc26705ac99a0c92efc19556d1e71c4fb484406b4295a51
Sep 09 20:38:56.492234 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:56.492214    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8mqdk_580818db-cd82-48dc-8ed9-4b7c631e9606/node-ca/0.log"
Sep 09 20:38:57.142347 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:57.142311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg" event={"ID":"0bc5ac3f-1d1b-48ea-835a-058b59debfbd","Type":"ContainerStarted","Data":"eca1c22e85d0dda85a89f0b5b0d1e29d9b0540c0ae39ea6582e92b32886006f4"}
Sep 09 20:38:57.142827 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:57.142365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg" event={"ID":"0bc5ac3f-1d1b-48ea-835a-058b59debfbd","Type":"ContainerStarted","Data":"ffa19728f66741c9edc26705ac99a0c92efc19556d1e71c4fb484406b4295a51"}
Sep 09 20:38:57.142827 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:57.142446    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:38:57.156961 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:57.156907    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg" podStartSLOduration=2.156887172 podStartE2EDuration="2.156887172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:57.156215543 +0000 UTC m=+1671.532342287" watchObservedRunningTime="2026-09-09 20:38:57.156887172 +0000 UTC m=+1671.533013919"
Sep 09 20:38:57.623667 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:57.623640    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ccvjh_f161b80c-3b9a-4256-82d1-b3dae8a7a5d6/serve-healthcheck-canary/0.log"
Sep 09 20:38:58.111865 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:58.111781    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sq74v_e4a8601a-75a0-4ff6-9937-f757ca1bc286/kube-rbac-proxy/0.log"
Sep 09 20:38:58.131871 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:58.131843    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sq74v_e4a8601a-75a0-4ff6-9937-f757ca1bc286/exporter/0.log"
Sep 09 20:38:58.150655 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:38:58.150631    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sq74v_e4a8601a-75a0-4ff6-9937-f757ca1bc286/extractor/0.log"
Sep 09 20:39:00.447474 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:00.447443    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-svxcw_a258b9d8-420e-4d37-82ca-3ffe8618a940/manager/0.log"
Sep 09 20:39:03.155518 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:03.155492    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6t7hr/perf-node-gather-daemonset-nljzg"
Sep 09 20:39:06.035291 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:06.035260    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fd985_a49fb02e-529d-46c8-9f79-3f023c94f54a/kube-multus-additional-cni-plugins/0.log"
Sep 09 20:39:06.053193 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:06.053165    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fd985_a49fb02e-529d-46c8-9f79-3f023c94f54a/egress-router-binary-copy/0.log"
Sep 09 20:39:06.071298 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:06.071276    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fd985_a49fb02e-529d-46c8-9f79-3f023c94f54a/cni-plugins/0.log"
Sep 09 20:39:06.088847 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:06.088825    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fd985_a49fb02e-529d-46c8-9f79-3f023c94f54a/bond-cni-plugin/0.log"
Sep 09 20:39:06.106403 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:06.106381    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fd985_a49fb02e-529d-46c8-9f79-3f023c94f54a/routeoverride-cni/0.log"
Sep 09 20:39:06.124258 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:06.124234    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fd985_a49fb02e-529d-46c8-9f79-3f023c94f54a/whereabouts-cni-bincopy/0.log"
Sep 09 20:39:06.142413 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:06.142350    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fd985_a49fb02e-529d-46c8-9f79-3f023c94f54a/whereabouts-cni/0.log"
Sep 09 20:39:06.338998 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:06.338970    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nmhdg_0f86202c-71cc-4ec5-990b-71f77d5565cf/kube-multus/0.log"
Sep 09 20:39:06.465064 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:06.465038    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v2fjq_cd1b99c8-4157-4934-b9e3-1b7f51835443/network-metrics-daemon/0.log"
Sep 09 20:39:06.483552 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:06.483531    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v2fjq_cd1b99c8-4157-4934-b9e3-1b7f51835443/kube-rbac-proxy/0.log"
Sep 09 20:39:07.655338 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:07.655303    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk4kx_8f9801ea-4417-4cd9-9c0f-afde235d5fe5/ovn-controller/0.log"
Sep 09 20:39:07.673874 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:07.673847    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk4kx_8f9801ea-4417-4cd9-9c0f-afde235d5fe5/ovn-acl-logging/0.log"
Sep 09 20:39:07.699353 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:07.699327    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk4kx_8f9801ea-4417-4cd9-9c0f-afde235d5fe5/kube-rbac-proxy-node/0.log"
Sep 09 20:39:07.724075 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:07.724048    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk4kx_8f9801ea-4417-4cd9-9c0f-afde235d5fe5/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 20:39:07.745017 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:07.744996    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk4kx_8f9801ea-4417-4cd9-9c0f-afde235d5fe5/northd/0.log"
Sep 09 20:39:07.767221 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:07.767198    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk4kx_8f9801ea-4417-4cd9-9c0f-afde235d5fe5/nbdb/0.log"
Sep 09 20:39:07.788555 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:07.788530    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk4kx_8f9801ea-4417-4cd9-9c0f-afde235d5fe5/sbdb/0.log"
Sep 09 20:39:07.825283 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:07.825267    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rk4kx_8f9801ea-4417-4cd9-9c0f-afde235d5fe5/ovnkube-controller/0.log"
Sep 09 20:39:08.800783 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:08.800752    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-7clz9_16520da4-678a-4a74-bb1e-6931c60e0f75/network-check-target-container/0.log"
Sep 09 20:39:09.731102 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:09.731073    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-ch7rb_af182f8e-b54c-44fe-b03e-69beceef79b9/iptables-alerter/0.log"
Sep 09 20:39:10.327206 ip-10-0-132-109 kubenswrapper[2579]: I0909 20:39:10.327176    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-8jbnx_8fcf2658-bdaa-4a8d-ae96-c4ca6b95426a/tuned/0.log"